Skip to content

fix(ci): validate the registry manifest offline, deterministically - #142

Merged
cdeust merged 2 commits into
mainfrom
agent/ci-offline-registry-validate
Aug 10, 2026
Merged

fix(ci): validate the registry manifest offline, deterministically#142
cdeust merged 2 commits into
mainfrom
agent/ci-offline-registry-validate

Conversation

@cdeust

@cdeust cdeust commented Aug 10, 2026

Copy link
Copy Markdown
Owner

Reopens the work of #141, which GitHub closed automatically when its stacked base branch was deleted on merge of #140. Same branch, same commits, retargeted to main.

The defect

.github/workflows/ci.yml's "Validate official MCP Registry manifest" step runs mcp-publisher validate, which POSTs to https://registry.modelcontextprotocol.io/v0/validate. It is a required status check that depends on a live third party.

Measured on PR #139: both test jobs failed with dial tcp 34.61.200.254:443: i/o timeout while 1335 tests passed. Re-running turned them green. It was not the diff — it is a required check that fails whenever a third party is briefly unreachable, and it will keep doing so.

Deliberately not fixed with a retry loop. A retry is a wall-clock verdict with nothing to synchronise on; here a deterministic alternative exists.

What /v0/validate actually does, read at the source

From modelcontextprotocol/registry at tag v1.8.1, not from a summary:

  • internal/api/handlers/v0/validate.go:26 calls ValidateServerJSON(server, ValidationAll) — full JSON Schema draft-07 structural validation plus Go-only semantic rules in internal/validators/validators.go: name and version format, per-source repository URL rules, https-only website and icon URLs, package-argument and transport-template-variable resolution.
  • The schema is //go:embed'd into the binary (internal/validators/schema.go), not fetched per request — so that half is a pure function, reproducible offline.
  • internal/api/handlers/v0/publish.go:58 already runs the same semantic checks (ValidationSchemaVersionAndSemantic) authoritatively at release time, in Release.yaml's publish-registry job.

So dropping the semantic pre-flight from CI moves detection later in the pipeline; it does not remove it.

What the replacement covers, and what it does not

scripts/validate_server_manifest.py runs jsonschema.Draft7Validator against a vendored, byte-identical copy of the registry's own embedded schema (scripts/schemas/2025-12-11.json, provenance recorded in scripts/schemas/README.md). Same input, same schema, spec-conformant validator: identical result to the structural half.

format_checker is deliberately omitted, because the Go compiler never sets AssertFormat — it defaults to false in santhosh-tekuri/jsonschema/v5. Enabling it here would make CI stricter than the endpoint it stands in for, which is its own kind of wrong answer.

It does not cover the Go-only semantic rules. Those remain covered by the authoritative live check at publish time, plus a new continue-on-error live mcp-publisher validate step that gives early warning when the registry happens to be reachable and cannot fail the build when it is not.

Gate

With HTTP(S)_PROXY pointed at an unreachable port, so any network attempt fails instantly:

  • a deliberately broken manifest (no / in name, missing version) is rejected, with both violations named;
  • the real server.json passes.

Identical outcome whether or not the network is reachable — which is the property the previous step lacked.

Reuses the composite action from #140 rather than introducing a third mcp-publisher pin.

ci.yml's "Validate official MCP Registry manifest" step downloaded
mcp-publisher and POSTed server.json to the live
registry.modelcontextprotocol.io/v0/validate endpoint, as a REQUIRED
check. Measured on PR #139: both `test` jobs failed with `dial tcp
34.61.200.254:443: i/o timeout` while 1335 tests passed; re-running turned
them green, and main shows four consecutive successes since. Not the
diff -- a required check that fails whenever a third party is briefly
unreachable, indefinitely.

Read the registry's own source (modelcontextprotocol/registry, tag
v1.8.1) to establish what /v0/validate actually checks before removing
the network call, per the standing "verify upstream claims at the
source" rule:

  internal/api/handlers/v0/validate.go:26 calls
  ValidateServerJSON(server, ValidationAll) -- full JSON Schema (draft-07)
  structural validation, PLUS Go-only semantic rules (name/version format,
  repository/website/icon URL rules, package argument and transport
  template-variable resolution) -- internal/validators/validators.go.

  The schema half is a pure function of server.json + a static schema
  document the registry binary embeds via `//go:embed schemas/*.json`
  (internal/validators/schema.go) rather than fetches per request --
  reproducible offline, byte-for-byte, with any spec-conformant draft-07
  validator, because both implementations run the identical specification
  over the identical input.

  The semantic half has no equivalent expressible in the JSON Schema
  document and is Go-only. It is not silently dropped from the release
  pipeline: internal/api/handlers/v0/publish.go:58 runs the SAME semantic
  checks -- ValidateServerJSON(server, ValidationSchemaVersionAndSemantic)
  -- authoritatively, over the network, at actual release time
  (Release.yaml's publish-registry job), and fails the release loudly if
  violated.

Fix:
- scripts/validate_server_manifest.py -- offline validator using
  `jsonschema`'s Draft7Validator against a vendored copy of the exact
  schema file the registry embeds (scripts/schemas/2025-12-11.json,
  scripts/schemas/README.md documents provenance + re-vendor procedure).
  Deliberately does NOT enable jsonschema's format_checker: the Go
  compiler in schema.go never sets `AssertFormat` (defaults false in
  santhosh-tekuri/jsonschema/v5), so `format: uri` is annotation-only
  there -- matching that, rather than being stricter than the real
  endpoint, avoids rejecting a server.json the registry would accept.
  Fails loudly (not silently-stale) if server.json's $schema ever moves
  to a version this repo hasn't re-vendored.
- ci.yml: this offline check is now the REQUIRED step. A second,
  `continue-on-error: true` step still calls live `mcp-publisher validate`
  for the semantic-only checks the offline step cannot cover -- early
  warning in the PR checks list when the registry is reachable, never a
  build failure when it isn't. Uses the shared
  .github/actions/install-mcp-publisher composite action from the
  preceding PR (agent/ci-mcp-publisher-dedup) rather than a third pin.

Verified: ruff check + format clean; YAML parses; actionlint clean on
ci.yml. Adversarial gate, offline (HTTP(S)_PROXY pointed at an
unreachable port so any network attempt fails instantly): a deliberately
broken server.json (no `/` in name, missing required `version`) is
rejected with both violations named; the real, unmodified server.json
passes; both outcomes identical whether or not the network is reachable.

Co-Authored-By: Claude <noreply@anthropic.com>
@cdeust
cdeust force-pushed the agent/ci-offline-registry-validate branch from 98049fc to 5fa991a Compare August 10, 2026 17:48
… add automated drift check

Answering the review question on this branch's schema vendoring: what
prevents the vendored copy from drifting silently out of sync with its
source? Documented provenance is the minimum, not a guarantee -- so this
adds an automatable check rather than leaving it to discipline.

While building it, the check found real drift. The vendored file was
first pulled from modelcontextprotocol/registry's v1.8.1 git tag
(internal/validators/schemas/2025-12-11.json), on the assumption that a
dated schema version is immutable once published. It measurably was not:
diffed against modelcontextprotocol/static (the schema's actual origin)
and the CDN it publishes to (https://static.modelcontextprotocol.io/...
-- the exact URL server.json's $schema field points to), the registry-tag
copy carried an extra `maxLength: 255` constraint on Package.version that
neither of the other two sources has. Root cause: the registry repo's own
sync-schema.yml is workflow_dispatch-only ("TODO: Add daily schedule
later"), so its embedded copy can lag the source it mirrors.

Fix:
- Re-vendored scripts/schemas/2025-12-11.json directly from the CDN
  (ground truth -- the URL server.json's $schema literally references),
  not the registry repo tag. Net diff: one stray constraint removed.
- scripts/check_vendored_schema_drift.py -- fetches the vendored file's
  own $id (which IS the canonical URL) and compares structurally.
  Wired into ci.yml as a third non-blocking (`continue-on-error`) step,
  same trade as the live semantic pre-flight already in this job: signal
  when reachable, never a required-check failure when it isn't. The
  offline structural check earlier in the job remains the only gate.
- scripts/schemas/README.md rewritten: CDN as the documented source of
  truth (not the registry repo tag), the drift that was found, and how to
  run the check standalone.

Verified: ruff check + format clean on scripts/; YAML parses; actionlint
clean. Drift check exercised three ways -- passes against the corrected
vendored file, fails with a named diagnosis when the removed maxLength
constraint is reintroduced, and fails loudly (nonzero exit, not a silent
pass) when the network is unreachable. Offline structural-validation gate
re-run against the corrected schema: unaffected (server.json's version
string is far under either bound).

Co-Authored-By: Claude <noreply@anthropic.com>
@cdeust

cdeust commented Aug 10, 2026

Copy link
Copy Markdown
Owner Author

Rebased onto the post-squash `main` (`96c45ff`) after #140 merged — dropped the now-redundant dedup commit, confirmed `.github/actions/install-mcp-publisher/action.yml` is identical to `main`'s (no reintroduced third pin: `git diff origin/main -- .github/actions/install-mcp-publisher/action.yml` is empty), and force-pushed. All checks (YAML parse, actionlint, ruff, the offline adversarial demo with an unreachable proxy) re-run clean post-rebase.

Answering the schema-drift question directly (added as a second commit, `85894d3`): documented provenance alone doesn't prevent silent drift, so this adds an automated check rather than relying on discipline — `scripts/check_vendored_schema_drift.py` fetches the vendored file's own `$id` (the canonical CDN URL, same one `server.json`'s `$schema` points to) and compares structurally. Wired into `ci.yml` as a third non-blocking step, same trade as the live semantic pre-flight already in this PR.

Building it surfaced real drift: the schema was first vendored from the `modelcontextprotocol/registry` git tag `v1.8.1`, on the assumption a dated schema is immutable once published. It wasn't — diffed against `modelcontextprotocol/static` (the schema's actual origin) and the CDN, the registry-tag copy carried a stray `maxLength: 255` on `Package.version` that neither other source has. Cause: the registry repo's own `sync-schema.yml` is `workflow_dispatch`-only, no schedule, so it can lag. Re-vendored from the CDN directly (documented as the source of truth now, not the registry tag) — one-line diff, no functional effect on this repo's `server.json` either way.

@cdeust

cdeust commented Aug 10, 2026

Copy link
Copy Markdown
Owner Author

ZETETIC-REVIEW: APPROVE

A required check that calls a live third party will fail whenever that party blinks, and it did: on #139 both test jobs died on dial tcp ... i/o timeout while 1335 tests passed. Fixing that with a retry was explicitly off the table — a retry is a wall-clock verdict with nothing to synchronise on. This removes the network instead.

The decision rests on reading the registry's source, not its documentation. At tag v1.8.1: validate.go:26 runs ValidateServerJSON(server, ValidationAll) — draft-07 structural validation plus Go-only semantic rules with no schema equivalent — and the schema is //go:embed'd rather than fetched, so the structural half is a pure function reproducible offline. The finding that decides the trade is publish.go:58: the same semantic checks already run authoritatively at publish time. So CI dropping the pre-flight moves detection later in the pipeline rather than out of it. That is stated rather than glossed, along with what is not covered.

Refusing to be stricter than the thing it replaces. format_checker is deliberately omitted because the Go compiler never sets AssertFormat — it defaults to false in santhosh-tekuri/jsonschema/v5. Enabling it would have failed manifests the real endpoint accepts. A substitute that rejects more than the original is wrong in the direction nobody tests for.

The gate is adversarial and offline. With HTTP(S)_PROXY pointed at an unreachable port so any network attempt fails instantly: a deliberately broken manifest (no / in name, missing version) is rejected with both violations named, and the real server.json passes. Identical outcome network or no network — the property the previous step lacked.

On drift, the answer was better than my question. I asked what stops a vendored schema copy from drifting, expecting either an automated check or an honest admission of a documented limit. The answer is the first, and it paid for itself before merging: the vendored file's own $id is the canonical URL, so check_vendored_schema_drift.py fetches and compares structurally — and running it found real drift already present. The copy taken from the registry's git tag carried a maxLength: 255 on Package.version that neither modelcontextprotocol/static nor the CDN has, because the registry's own sync-schema.yml is workflow_dispatch-only with a "TODO: Add daily schedule later". The vendored copy was wrong on arrival, re-vendored from the CDN, and the source of truth corrected in scripts/schemas/README.md.

The drift check is continue-on-error, matching the trade already made for the live semantic pre-flight: signal when reachable, never a required failure when it is not — and it fails loudly rather than passing silently when the network is down, which is the distinction that matters.

The rebase was verified, not assumed. git diff origin/main -- .github/actions/install-mcp-publisher/action.yml is empty, so no third pin was reintroduced — the exact hazard the stacked split existed to avoid. All short checks re-ran after the rebase, including the adversarial demo, confirming no path moved silently.

Green and clean at 85894d3. Merging.

@cdeust
cdeust merged commit 7703702 into main Aug 10, 2026
12 checks passed
@cdeust
cdeust deleted the agent/ci-offline-registry-validate branch August 10, 2026 17:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant