fix(ci): validate the registry manifest offline, deterministically - #142
Conversation
ci.yml's "Validate official MCP Registry manifest" step downloaded mcp-publisher and POSTed server.json to the live registry.modelcontextprotocol.io/v0/validate endpoint, as a REQUIRED check. Measured on PR #139: both `test` jobs failed with `dial tcp 34.61.200.254:443: i/o timeout` while 1335 tests passed; re-running turned them green, and main shows four consecutive successes since. Not the diff -- a required check that fails whenever a third party is briefly unreachable, indefinitely. Read the registry's own source (modelcontextprotocol/registry, tag v1.8.1) to establish what /v0/validate actually checks before removing the network call, per the standing "verify upstream claims at the source" rule: internal/api/handlers/v0/validate.go:26 calls ValidateServerJSON(server, ValidationAll) -- full JSON Schema (draft-07) structural validation, PLUS Go-only semantic rules (name/version format, repository/website/icon URL rules, package argument and transport template-variable resolution) -- internal/validators/validators.go. The schema half is a pure function of server.json + a static schema document the registry binary embeds via `//go:embed schemas/*.json` (internal/validators/schema.go) rather than fetches per request -- reproducible offline, byte-for-byte, with any spec-conformant draft-07 validator, because both implementations run the identical specification over the identical input. The semantic half has no equivalent expressible in the JSON Schema document and is Go-only. It is not silently dropped from the release pipeline: internal/api/handlers/v0/publish.go:58 runs the SAME semantic checks -- ValidateServerJSON(server, ValidationSchemaVersionAndSemantic) -- authoritatively, over the network, at actual release time (Release.yaml's publish-registry job), and fails the release loudly if violated. Fix: - scripts/validate_server_manifest.py -- offline validator using `jsonschema`'s Draft7Validator against a vendored copy of the exact schema file the registry embeds (scripts/schemas/2025-12-11.json, scripts/schemas/README.md documents provenance + re-vendor procedure). Deliberately does NOT enable jsonschema's format_checker: the Go compiler in schema.go never sets `AssertFormat` (defaults false in santhosh-tekuri/jsonschema/v5), so `format: uri` is annotation-only there -- matching that, rather than being stricter than the real endpoint, avoids rejecting a server.json the registry would accept. Fails loudly (not silently-stale) if server.json's $schema ever moves to a version this repo hasn't re-vendored. - ci.yml: this offline check is now the REQUIRED step. A second, `continue-on-error: true` step still calls live `mcp-publisher validate` for the semantic-only checks the offline step cannot cover -- early warning in the PR checks list when the registry is reachable, never a build failure when it isn't. Uses the shared .github/actions/install-mcp-publisher composite action from the preceding PR (agent/ci-mcp-publisher-dedup) rather than a third pin. Verified: ruff check + format clean; YAML parses; actionlint clean on ci.yml. Adversarial gate, offline (HTTP(S)_PROXY pointed at an unreachable port so any network attempt fails instantly): a deliberately broken server.json (no `/` in name, missing required `version`) is rejected with both violations named; the real, unmodified server.json passes; both outcomes identical whether or not the network is reachable. Co-Authored-By: Claude <noreply@anthropic.com>
98049fc to
5fa991a
Compare
… add automated drift check Answering the review question on this branch's schema vendoring: what prevents the vendored copy from drifting silently out of sync with its source? Documented provenance is the minimum, not a guarantee -- so this adds an automatable check rather than leaving it to discipline. While building it, the check found real drift. The vendored file was first pulled from modelcontextprotocol/registry's v1.8.1 git tag (internal/validators/schemas/2025-12-11.json), on the assumption that a dated schema version is immutable once published. It measurably was not: diffed against modelcontextprotocol/static (the schema's actual origin) and the CDN it publishes to (https://static.modelcontextprotocol.io/... -- the exact URL server.json's $schema field points to), the registry-tag copy carried an extra `maxLength: 255` constraint on Package.version that neither of the other two sources has. Root cause: the registry repo's own sync-schema.yml is workflow_dispatch-only ("TODO: Add daily schedule later"), so its embedded copy can lag the source it mirrors. Fix: - Re-vendored scripts/schemas/2025-12-11.json directly from the CDN (ground truth -- the URL server.json's $schema literally references), not the registry repo tag. Net diff: one stray constraint removed. - scripts/check_vendored_schema_drift.py -- fetches the vendored file's own $id (which IS the canonical URL) and compares structurally. Wired into ci.yml as a third non-blocking (`continue-on-error`) step, same trade as the live semantic pre-flight already in this job: signal when reachable, never a required-check failure when it isn't. The offline structural check earlier in the job remains the only gate. - scripts/schemas/README.md rewritten: CDN as the documented source of truth (not the registry repo tag), the drift that was found, and how to run the check standalone. Verified: ruff check + format clean on scripts/; YAML parses; actionlint clean. Drift check exercised three ways -- passes against the corrected vendored file, fails with a named diagnosis when the removed maxLength constraint is reintroduced, and fails loudly (nonzero exit, not a silent pass) when the network is unreachable. Offline structural-validation gate re-run against the corrected schema: unaffected (server.json's version string is far under either bound). Co-Authored-By: Claude <noreply@anthropic.com>
|
Rebased onto the post-squash `main` (`96c45ff`) after #140 merged — dropped the now-redundant dedup commit, confirmed `.github/actions/install-mcp-publisher/action.yml` is identical to `main`'s (no reintroduced third pin: `git diff origin/main -- .github/actions/install-mcp-publisher/action.yml` is empty), and force-pushed. All checks (YAML parse, actionlint, ruff, the offline adversarial demo with an unreachable proxy) re-run clean post-rebase. Answering the schema-drift question directly (added as a second commit, `85894d3`): documented provenance alone doesn't prevent silent drift, so this adds an automated check rather than relying on discipline — `scripts/check_vendored_schema_drift.py` fetches the vendored file's own `$id` (the canonical CDN URL, same one `server.json`'s `$schema` points to) and compares structurally. Wired into `ci.yml` as a third non-blocking step, same trade as the live semantic pre-flight already in this PR. Building it surfaced real drift: the schema was first vendored from the `modelcontextprotocol/registry` git tag `v1.8.1`, on the assumption a dated schema is immutable once published. It wasn't — diffed against `modelcontextprotocol/static` (the schema's actual origin) and the CDN, the registry-tag copy carried a stray `maxLength: 255` on `Package.version` that neither other source has. Cause: the registry repo's own `sync-schema.yml` is `workflow_dispatch`-only, no schedule, so it can lag. Re-vendored from the CDN directly (documented as the source of truth now, not the registry tag) — one-line diff, no functional effect on this repo's `server.json` either way. |
|
ZETETIC-REVIEW: APPROVE A required check that calls a live third party will fail whenever that party blinks, and it did: on #139 both The decision rests on reading the registry's source, not its documentation. At tag Refusing to be stricter than the thing it replaces. The gate is adversarial and offline. With On drift, the answer was better than my question. I asked what stops a vendored schema copy from drifting, expecting either an automated check or an honest admission of a documented limit. The answer is the first, and it paid for itself before merging: the vendored file's own The drift check is The rebase was verified, not assumed. Green and clean at |
Reopens the work of #141, which GitHub closed automatically when its stacked base branch was deleted on merge of #140. Same branch, same commits, retargeted to
main.The defect
.github/workflows/ci.yml's "Validate official MCP Registry manifest" step runsmcp-publisher validate, which POSTs tohttps://registry.modelcontextprotocol.io/v0/validate. It is a required status check that depends on a live third party.Measured on PR #139: both
testjobs failed withdial tcp 34.61.200.254:443: i/o timeoutwhile 1335 tests passed. Re-running turned them green. It was not the diff — it is a required check that fails whenever a third party is briefly unreachable, and it will keep doing so.Deliberately not fixed with a retry loop. A retry is a wall-clock verdict with nothing to synchronise on; here a deterministic alternative exists.
What
/v0/validateactually does, read at the sourceFrom
modelcontextprotocol/registryat tagv1.8.1, not from a summary:internal/api/handlers/v0/validate.go:26callsValidateServerJSON(server, ValidationAll)— full JSON Schema draft-07 structural validation plus Go-only semantic rules ininternal/validators/validators.go: name and version format, per-source repository URL rules, https-only website and icon URLs, package-argument and transport-template-variable resolution.//go:embed'd into the binary (internal/validators/schema.go), not fetched per request — so that half is a pure function, reproducible offline.internal/api/handlers/v0/publish.go:58already runs the same semantic checks (ValidationSchemaVersionAndSemantic) authoritatively at release time, inRelease.yaml'spublish-registryjob.So dropping the semantic pre-flight from CI moves detection later in the pipeline; it does not remove it.
What the replacement covers, and what it does not
scripts/validate_server_manifest.pyrunsjsonschema.Draft7Validatoragainst a vendored, byte-identical copy of the registry's own embedded schema (scripts/schemas/2025-12-11.json, provenance recorded inscripts/schemas/README.md). Same input, same schema, spec-conformant validator: identical result to the structural half.format_checkeris deliberately omitted, because the Go compiler never setsAssertFormat— it defaults tofalseinsanthosh-tekuri/jsonschema/v5. Enabling it here would make CI stricter than the endpoint it stands in for, which is its own kind of wrong answer.It does not cover the Go-only semantic rules. Those remain covered by the authoritative live check at publish time, plus a new
continue-on-errorlivemcp-publisher validatestep that gives early warning when the registry happens to be reachable and cannot fail the build when it is not.Gate
With
HTTP(S)_PROXYpointed at an unreachable port, so any network attempt fails instantly:/inname, missingversion) is rejected, with both violations named;server.jsonpasses.Identical outcome whether or not the network is reachable — which is the property the previous step lacked.
Reuses the composite action from #140 rather than introducing a third
mcp-publisherpin.