fix(ci): stop requiring a live registry call to validate server.json - #141
Closed
cdeust wants to merge 1 commit into
Closed
fix(ci): stop requiring a live registry call to validate server.json#141cdeust wants to merge 1 commit into
cdeust wants to merge 1 commit into
Conversation
ci.yml's "Validate official MCP Registry manifest" step downloaded mcp-publisher and POSTed server.json to the live registry.modelcontextprotocol.io/v0/validate endpoint, as a REQUIRED check. Measured on PR #139: both `test` jobs failed with `dial tcp 34.61.200.254:443: i/o timeout` while 1335 tests passed; re-running turned them green, and main shows four consecutive successes since. Not the diff -- a required check that fails whenever a third party is briefly unreachable, indefinitely. Read the registry's own source (modelcontextprotocol/registry, tag v1.8.1) to establish what /v0/validate actually checks before removing the network call, per the standing "verify upstream claims at the source" rule: internal/api/handlers/v0/validate.go:26 calls ValidateServerJSON(server, ValidationAll) -- full JSON Schema (draft-07) structural validation, PLUS Go-only semantic rules (name/version format, repository/website/icon URL rules, package argument and transport template-variable resolution) -- internal/validators/validators.go. The schema half is a pure function of server.json + a static schema document the registry binary embeds via `//go:embed schemas/*.json` (internal/validators/schema.go) rather than fetches per request -- reproducible offline, byte-for-byte, with any spec-conformant draft-07 validator, because both implementations run the identical specification over the identical input. The semantic half has no equivalent expressible in the JSON Schema document and is Go-only. It is not silently dropped from the release pipeline: internal/api/handlers/v0/publish.go:58 runs the SAME semantic checks -- ValidateServerJSON(server, ValidationSchemaVersionAndSemantic) -- authoritatively, over the network, at actual release time (Release.yaml's publish-registry job), and fails the release loudly if violated. Fix: - scripts/validate_server_manifest.py -- offline validator using `jsonschema`'s Draft7Validator against a vendored copy of the exact schema file the registry embeds (scripts/schemas/2025-12-11.json, scripts/schemas/README.md documents provenance + re-vendor procedure). Deliberately does NOT enable jsonschema's format_checker: the Go compiler in schema.go never sets `AssertFormat` (defaults false in santhosh-tekuri/jsonschema/v5), so `format: uri` is annotation-only there -- matching that, rather than being stricter than the real endpoint, avoids rejecting a server.json the registry would accept. Fails loudly (not silently-stale) if server.json's $schema ever moves to a version this repo hasn't re-vendored. - ci.yml: this offline check is now the REQUIRED step. A second, `continue-on-error: true` step still calls live `mcp-publisher validate` for the semantic-only checks the offline step cannot cover -- early warning in the PR checks list when the registry is reachable, never a build failure when it isn't. Uses the shared .github/actions/install-mcp-publisher composite action from the preceding PR (agent/ci-mcp-publisher-dedup) rather than a third pin. Verified: ruff check + format clean; YAML parses; actionlint clean on ci.yml. Adversarial gate, offline (HTTP(S)_PROXY pointed at an unreachable port so any network attempt fails instantly): a deliberately broken server.json (no `/` in name, missing required `version`) is rejected with both violations named; the real, unmodified server.json passes; both outcomes identical whether or not the network is reachable. Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #140 — uses the shared
install-mcp-publishercompositeaction from that PR rather than adding a third pin. Merge #140 first.
Defect
ci.yml's "Validate official MCP Registry manifest" step downloadedmcp-publisherand POSTedserver.jsonto the liveregistry.modelcontextprotocol.io/v0/validate, as a required check.Measured on PR #139: both
testjobs failed withdial tcp 34.61.200.254:443: i/o timeoutwhile 1335 tests passed; re-running turnedthem green, and
mainshows four consecutive successes since. Not thediff — a required check that fails whenever a third party is briefly
unreachable, and will keep doing so.
What
/v0/validateactually checks (read at the source, not summarized)modelcontextprotocol/registrytagv1.8.1,internal/api/handlers/v0/validate.go:26:ValidateServerJSON(server, ValidationAll)= full JSON Schema (draft-07)structural validation plus Go-only semantic rules (name/version
format, repository/website/icon URL rules, package argument and
transport template-variable resolution) —
internal/validators/validators.go.The schema half is a pure function of
server.json+ a static schemadocument the registry binary embeds at build time
(
//go:embed schemas/*.json,internal/validators/schema.go) rather thanfetches per request. Reproducible offline, byte-for-byte, with any
spec-conformant draft-07 validator — both implementations run the
identical specification over the identical input.
The semantic half has no equivalent expressible in the JSON Schema
document. It is not silently dropped from the release pipeline:
internal/api/handlers/v0/publish.go:58runs the same semantic checks —ValidateServerJSON(server, ValidationSchemaVersionAndSemantic)—authoritatively, over the network, at actual release time
(
Release.yaml'spublish-registryjob), and fails the release loudlyif violated.
Fix
scripts/validate_server_manifest.py— offline validator,jsonschema'sDraft7Validatoragainst a vendored copy of the exact schema file theregistry embeds (
scripts/schemas/2025-12-11.json,scripts/schemas/README.mddocuments provenance + re-vendor procedure).Deliberately does not enable
jsonschema'sformat_checker: the Gocompiler never sets
AssertFormat(defaultsfalseinsanthosh-tekuri/jsonschema/v5), soformat: uriis annotation-only onthe real endpoint too — matching that avoids being stricter than the
registry and rejecting a manifest it would accept. Fails loudly (not
silently stale) if
server.json's$schemaever moves to a versionthis repo hasn't re-vendored.
ci.yml— the offline check is now the required step. A second,continue-on-error: truestep still calls livemcp-publisher validatefor the semantic-only checks the offline step can't cover: early
warning in the PR checks list when the registry is reachable, never a
build failure when it isn't.
Verification
ruff check+ruff format --checkon the new script: clean.actionlint .github/workflows/ci.yml: clean.HTTP(S)_PROXYpointed at anunreachable port, so any network attempt fails instantly instead of
hanging):
Co-Authored-By: Claude noreply@anthropic.com