ci: pin the conformance catalog by SHA instead of cloning its default branch - #18
ci: pin the conformance catalog by SHA instead of cloning its default branch#18RobertoIskandarani wants to merge 2 commits into
Conversation
… branch CI cloned github.com/AuthPlane/conformance at its default branch, so any change to the catalog reached this repo immediately — a case added there could turn an unrelated PR red here with nothing in this repo having changed. Pinning decouples them: a catalog change arrives only when this repo deliberately bumps the pin together with the coverage for it. The revision is single-sourced in a tracked .conformance-catalog-ref, guarded by a 40-hex shape check before the fetch so a branch or tag name cannot silently un-pin CI, and read by both ci.yml and release.yml. A weekly conformance-catalog-drift workflow clones the unpinned tip and runs the alignment assertion. Its report distinguishes real drift from an infrastructure failure — a clone that never landed, or a step that did not run — so a red run points at the right thing. No SDK source changes. Verified locally against the pinned revision: the catalog alignment assertion passes and the full suite is green (115 tests). The pin adopts no new case ids, so it requires no coverage change.
…ctions The pin decouples this repo from catalog changes, but nothing enforced that the pinned revision and the SDK's coverage actually agree — the weekly drift job runs against the unpinned tip, so a pin bump itself went unchecked. The assertion now runs in the normal suite, against the pinned catalog, in both directions: every catalog case has a mapping, and every mapping names a case the catalog has. The reverse direction found one on the first run. Rfc7662ConformanceTest registered `rfc9449-introspection-response-must-expose-cnf-jkt`, which no catalog revision defines. Results for an unknown id are silently dropped from the conformance report, since the report is built by iterating the catalog — so that test has been running and reporting nothing. The catalog's only cnf.jkt case, rfc9449-dpop-bound-token-must-contain-cnf-jkt, covers the verifier rejecting a DPoP-bound token that lacks the claim, and is already mapped elsewhere. That is a different requirement from the introspection response exposing the thumbprint, so the annotation is dropped rather than repointed, and the test stays as SDK-side coverage of RFC 9449 §6.2 Figure 11, reported under uncatalogued tests. Also documents the pin for contributors: how to check the catalog out at the same ref CI uses, and that adopting new cases means bumping the ref in the same change. Verified against the pinned revision: the alignment assertion passes in both directions and the full suite is green (115 tests).
|
Added Pinning decouples this repo from catalog changes, but nothing was checking that The reverse direction found one on the first run
That is not cosmetic. The conformance report is built by iterating the catalog, so The catalog's only Also
VerificationAgainst the pinned revision: the alignment assertion passes in both directions, |
What
CI clones
github.com/AuthPlane/conformanceat its default branch, so any changeto the catalog reaches this repo immediately. A case added there can turn an
unrelated PR red here with nothing in this repo having changed.
This pins the revision instead. A catalog change arrives only when this repo
deliberately bumps the pin together with the coverage for it.
.conformance-catalog-ref— a tracked file holding a 40-hex commit SHA,guarded by a shape check before the fetch so a branch or tag name cannot
silently un-pin CI.
ci.yml/release.yml— both read that file instead of cloning the tip..github/workflows/conformance-catalog-drift.yml— a weekly job that clonesthe unpinned tip and runs the alignment assertion, so new cases surface as an
early warning rather than a surprise at bump time.
The drift job's report separates real drift from an infrastructure failure — a
clone that never landed, or an alignment step that never ran — so a red run points
at the right cause instead of sending the reader to hunt for uncovered cases that
do not exist.
Scope
No SDK source is touched — four files, all CI.
Verification
Run locally against the pinned revision: the catalog alignment assertion passes
and the full suite is green (115 tests). The pin adopts no new case ids, so it
requires no coverage change.
actionlintclean on both modified workflows.Known follow-up, deliberately not fixed here
The read/guard/fetch block is duplicated in
ci.ymlandrelease.yml. The refis single-sourced; the ten lines reading it are not, so the guard could be
tightened in one workflow and not the other.
Extracting it to a shared script is the right fix and has already been done in the
Go repository. It is left out on purpose: doing it here first would put this copy
ahead of its private counterpart and create drift in the opposite direction. The
extraction should land there and port, so the two stay identical.