Skip to content

fix(ci): stop requiring a live registry call to validate server.json - #141

Closed
cdeust wants to merge 1 commit into
agent/ci-mcp-publisher-dedupfrom
agent/ci-offline-registry-validate
Closed

fix(ci): stop requiring a live registry call to validate server.json#141
cdeust wants to merge 1 commit into
agent/ci-mcp-publisher-dedupfrom
agent/ci-offline-registry-validate

Conversation

@cdeust

@cdeust cdeust commented Aug 10, 2026

Copy link
Copy Markdown
Owner

Stacked on #140 — uses the shared install-mcp-publisher composite
action from that PR rather than adding a third pin. Merge #140 first.

Defect

ci.yml's "Validate official MCP Registry manifest" step downloaded
mcp-publisher and POSTed server.json to the live
registry.modelcontextprotocol.io/v0/validate, as a required check.
Measured on PR #139: both test jobs failed with dial tcp 34.61.200.254:443: i/o timeout while 1335 tests passed; re-running turned
them green, and main shows four consecutive successes since. Not the
diff — a required check that fails whenever a third party is briefly
unreachable, and will keep doing so.

What /v0/validate actually checks (read at the source, not summarized)

modelcontextprotocol/registry tag v1.8.1,
internal/api/handlers/v0/validate.go:26:
ValidateServerJSON(server, ValidationAll) = full JSON Schema (draft-07)
structural validation plus Go-only semantic rules (name/version
format, repository/website/icon URL rules, package argument and
transport template-variable resolution) —
internal/validators/validators.go.

The schema half is a pure function of server.json + a static schema
document the registry binary embeds at build time
(//go:embed schemas/*.json, internal/validators/schema.go) rather than
fetches per request. Reproducible offline, byte-for-byte, with any
spec-conformant draft-07 validator — both implementations run the
identical specification over the identical input.

The semantic half has no equivalent expressible in the JSON Schema
document. It is not silently dropped from the release pipeline:
internal/api/handlers/v0/publish.go:58 runs the same semantic checks —
ValidateServerJSON(server, ValidationSchemaVersionAndSemantic)
authoritatively, over the network, at actual release time
(Release.yaml's publish-registry job), and fails the release loudly
if violated.

Fix

  • scripts/validate_server_manifest.py — offline validator, jsonschema's
    Draft7Validator against a vendored copy of the exact schema file the
    registry embeds (scripts/schemas/2025-12-11.json,
    scripts/schemas/README.md documents provenance + re-vendor procedure).
    Deliberately does not enable jsonschema's format_checker: the Go
    compiler never sets AssertFormat (defaults false in
    santhosh-tekuri/jsonschema/v5), so format: uri is annotation-only on
    the real endpoint too — matching that avoids being stricter than the
    registry and rejecting a manifest it would accept. Fails loudly (not
    silently stale) if server.json's $schema ever moves to a version
    this repo hasn't re-vendored.
  • ci.yml — the offline check is now the required step. A second,
    continue-on-error: true step still calls live mcp-publisher validate
    for the semantic-only checks the offline step can't cover: early
    warning in the PR checks list when the registry is reachable, never a
    build failure when it isn't.

Verification

  • ruff check + ruff format --check on the new script: clean.
  • YAML parses; actionlint .github/workflows/ci.yml: clean.
  • Adversarial gate, offline (HTTP(S)_PROXY pointed at an
    unreachable port, so any network attempt fails instantly instead of
    hanging):
    === real manifest, network unreachable ===
    server.json is schema-valid against https://static.modelcontextprotocol.io/schemas/2025-12-11/server.schema.json
    exit: 0
    
    === broken manifest (no '/' in name, missing required 'version'), network unreachable ===
    server.json FAILED schema validation with 2 issue(s):
    1. $: 'version' is a required property
    2. $['name']: 'no-slash-name' does not match '^[a-zA-Z0-9.-]+/[a-zA-Z0-9._-]+$'
    exit code: 1
    

Co-Authored-By: Claude noreply@anthropic.com

ci.yml's "Validate official MCP Registry manifest" step downloaded
mcp-publisher and POSTed server.json to the live
registry.modelcontextprotocol.io/v0/validate endpoint, as a REQUIRED
check. Measured on PR #139: both `test` jobs failed with `dial tcp
34.61.200.254:443: i/o timeout` while 1335 tests passed; re-running turned
them green, and main shows four consecutive successes since. Not the
diff -- a required check that fails whenever a third party is briefly
unreachable, indefinitely.

Read the registry's own source (modelcontextprotocol/registry, tag
v1.8.1) to establish what /v0/validate actually checks before removing
the network call, per the standing "verify upstream claims at the
source" rule:

  internal/api/handlers/v0/validate.go:26 calls
  ValidateServerJSON(server, ValidationAll) -- full JSON Schema (draft-07)
  structural validation, PLUS Go-only semantic rules (name/version format,
  repository/website/icon URL rules, package argument and transport
  template-variable resolution) -- internal/validators/validators.go.

  The schema half is a pure function of server.json + a static schema
  document the registry binary embeds via `//go:embed schemas/*.json`
  (internal/validators/schema.go) rather than fetches per request --
  reproducible offline, byte-for-byte, with any spec-conformant draft-07
  validator, because both implementations run the identical specification
  over the identical input.

  The semantic half has no equivalent expressible in the JSON Schema
  document and is Go-only. It is not silently dropped from the release
  pipeline: internal/api/handlers/v0/publish.go:58 runs the SAME semantic
  checks -- ValidateServerJSON(server, ValidationSchemaVersionAndSemantic)
  -- authoritatively, over the network, at actual release time
  (Release.yaml's publish-registry job), and fails the release loudly if
  violated.

Fix:
- scripts/validate_server_manifest.py -- offline validator using
  `jsonschema`'s Draft7Validator against a vendored copy of the exact
  schema file the registry embeds (scripts/schemas/2025-12-11.json,
  scripts/schemas/README.md documents provenance + re-vendor procedure).
  Deliberately does NOT enable jsonschema's format_checker: the Go
  compiler in schema.go never sets `AssertFormat` (defaults false in
  santhosh-tekuri/jsonschema/v5), so `format: uri` is annotation-only
  there -- matching that, rather than being stricter than the real
  endpoint, avoids rejecting a server.json the registry would accept.
  Fails loudly (not silently-stale) if server.json's $schema ever moves
  to a version this repo hasn't re-vendored.
- ci.yml: this offline check is now the REQUIRED step. A second,
  `continue-on-error: true` step still calls live `mcp-publisher validate`
  for the semantic-only checks the offline step cannot cover -- early
  warning in the PR checks list when the registry is reachable, never a
  build failure when it isn't. Uses the shared
  .github/actions/install-mcp-publisher composite action from the
  preceding PR (agent/ci-mcp-publisher-dedup) rather than a third pin.

Verified: ruff check + format clean; YAML parses; actionlint clean on
ci.yml. Adversarial gate, offline (HTTP(S)_PROXY pointed at an
unreachable port so any network attempt fails instantly): a deliberately
broken server.json (no `/` in name, missing required `version`) is
rejected with both violations named; the real, unmodified server.json
passes; both outcomes identical whether or not the network is reachable.

Co-Authored-By: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant