Skip to content

ci: pin the conformance catalog by SHA instead of cloning its default branch - #18

Open
RobertoIskandarani wants to merge 2 commits into
mainfrom
port/pin-conformance-catalog
Open

ci: pin the conformance catalog by SHA instead of cloning its default branch#18
RobertoIskandarani wants to merge 2 commits into
mainfrom
port/pin-conformance-catalog

Conversation

@RobertoIskandarani

Copy link
Copy Markdown
Collaborator

What

CI clones github.com/AuthPlane/conformance at its default branch, so any change
to the catalog reaches this repo immediately. A case added there can turn an
unrelated PR red here with nothing in this repo having changed.

This pins the revision instead. A catalog change arrives only when this repo
deliberately bumps the pin together with the coverage for it.

  • .conformance-catalog-ref — a tracked file holding a 40-hex commit SHA,
    guarded by a shape check before the fetch so a branch or tag name cannot
    silently un-pin CI.
  • ci.yml / release.yml — both read that file instead of cloning the tip.
  • .github/workflows/conformance-catalog-drift.yml — a weekly job that clones
    the unpinned tip and runs the alignment assertion, so new cases surface as an
    early warning rather than a surprise at bump time.

The drift job's report separates real drift from an infrastructure failure — a
clone that never landed, or an alignment step that never ran — so a red run points
at the right cause instead of sending the reader to hunt for uncovered cases that
do not exist.

Scope

No SDK source is touched — four files, all CI.

Verification

Run locally against the pinned revision: the catalog alignment assertion passes
and the full suite is green (115 tests). The pin adopts no new case ids, so it
requires no coverage change. actionlint clean on both modified workflows.

Known follow-up, deliberately not fixed here

The read/guard/fetch block is duplicated in ci.yml and release.yml. The ref
is single-sourced; the ten lines reading it are not, so the guard could be
tightened in one workflow and not the other.

Extracting it to a shared script is the right fix and has already been done in the
Go repository. It is left out on purpose: doing it here first would put this copy
ahead of its private counterpart and create drift in the opposite direction. The
extraction should land there and port, so the two stay identical.

… branch

CI cloned github.com/AuthPlane/conformance at its default branch, so any change
to the catalog reached this repo immediately — a case added there could turn an
unrelated PR red here with nothing in this repo having changed. Pinning
decouples them: a catalog change arrives only when this repo deliberately bumps
the pin together with the coverage for it.

The revision is single-sourced in a tracked .conformance-catalog-ref, guarded by
a 40-hex shape check before the fetch so a branch or tag name cannot silently
un-pin CI, and read by both ci.yml and release.yml.

A weekly conformance-catalog-drift workflow clones the unpinned tip and runs the
alignment assertion. Its report distinguishes real drift from an infrastructure
failure — a clone that never landed, or a step that did not run — so a red run
points at the right thing.

No SDK source changes. Verified locally against the pinned revision: the catalog
alignment assertion passes and the full suite is green (115 tests). The pin
adopts no new case ids, so it requires no coverage change.
@RobertoIskandarani
RobertoIskandarani requested a review from a team as a code owner August 10, 2026 15:02
…ctions

The pin decouples this repo from catalog changes, but nothing enforced that the
pinned revision and the SDK's coverage actually agree — the weekly drift job runs
against the unpinned tip, so a pin bump itself went unchecked. The assertion now
runs in the normal suite, against the pinned catalog, in both directions: every
catalog case has a mapping, and every mapping names a case the catalog has.

The reverse direction found one on the first run.
Rfc7662ConformanceTest registered `rfc9449-introspection-response-must-expose-cnf-jkt`,
which no catalog revision defines. Results for an unknown id are silently dropped
from the conformance report, since the report is built by iterating the catalog —
so that test has been running and reporting nothing.

The catalog's only cnf.jkt case, rfc9449-dpop-bound-token-must-contain-cnf-jkt,
covers the verifier rejecting a DPoP-bound token that lacks the claim, and is
already mapped elsewhere. That is a different requirement from the introspection
response exposing the thumbprint, so the annotation is dropped rather than
repointed, and the test stays as SDK-side coverage of RFC 9449 §6.2 Figure 11,
reported under uncatalogued tests.

Also documents the pin for contributors: how to check the catalog out at the same
ref CI uses, and that adopting new cases means bumping the ref in the same change.

Verified against the pinned revision: the alignment assertion passes in both
directions and the full suite is green (115 tests).
@RobertoIskandarani

Copy link
Copy Markdown
Collaborator Author

Added bb1f97a — the enforcement half of the pin, which the first push left out.

Pinning decouples this repo from catalog changes, but nothing was checking that
the pinned revision and the SDK's coverage actually agree. The weekly drift job
runs against the unpinned tip, so the pin bump itself went unchecked — exactly
the gap that lets a bump land quietly wrong. The assertion now runs in the normal
suite, against the pinned catalog, in both directions: every catalog case has
a mapping, and every mapping names a case the catalog has.

The reverse direction found one on the first run

Rfc7662ConformanceTest registered rfc9449-introspection-response-must-expose-cnf-jkt.
No catalog revision defines it — not the pinned one, not the tip.

That is not cosmetic. The conformance report is built by iterating the catalog, so
results for an unknown id are silently dropped. That test has been running and
reporting nothing, and the one-directional check could never have said so.

The catalog's only cnf.jkt case — rfc9449-dpop-bound-token-must-contain-cnf-jkt
— covers the verifier rejecting a DPoP-bound token that lacks the claim, and is
already mapped in Rfc9449ConformanceTest. That is a different requirement from
the introspection response exposing the thumbprint, so the annotation is dropped
rather than repointed. The test stays as SDK-side coverage of RFC 9449 §6.2
Figure 11 and is reported under uncatalogued tests.

Also

CONTRIBUTING.md and core/src/conformance/README.md now say how to check the
catalog out at the same ref CI pins, and that adopting new cases means bumping the
ref in the same change. A pin nobody can reproduce locally just moves the surprise.

Verification

Against the pinned revision: the alignment assertion passes in both directions,
and the full suite is green — 115 tests, BUILD SUCCESS. The failure above was
observed before the fix, so the assertion demonstrably catches what it is for.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant