Skip to content

ci: pin the conformance catalog by SHA instead of cloning its default branch - #25

Open
RobertoIskandarani wants to merge 2 commits into
mainfrom
port/pin-conformance-catalog
Open

ci: pin the conformance catalog by SHA instead of cloning its default branch#25
RobertoIskandarani wants to merge 2 commits into
mainfrom
port/pin-conformance-catalog

Conversation

@RobertoIskandarani

Copy link
Copy Markdown

What

CI clones github.com/AuthPlane/conformance at its default branch, so any change
to the catalog reaches this repo immediately. A case added there can turn an
unrelated PR red here with nothing in this repo having changed.

This pins the revision instead. A catalog change arrives only when this repo
deliberately bumps the pin together with the coverage for it.

  • .conformance-catalog-ref — a tracked file holding a 40-hex commit SHA,
    guarded by a ^[0-9a-f]{40}$ shape check before the fetch so a branch or tag
    name cannot silently un-pin CI.
  • .github/scripts/fetch-conformance-catalog.sh — the read/guard/fetch
    sequence, called by both ci.yml and release.yml. The ref was already going
    to be single-sourced; the ~10 lines reading it would have been copy-pasted into
    two workflows and could tighten in one and not the other.
  • .github/workflows/conformance-catalog-drift.yml — a weekly job that clones
    the unpinned tip and runs the alignment assertion, so new cases surface as an
    early warning rather than a surprise at bump time.

The catalog now lands in $RUNNER_TEMP rather than the workspace, so it stays out
of go list ./..., coverage globs, and git add -A in the release commit.

Two details worth calling out

shell: bash on the alignment step is load-bearing. The default shell for a
run: step on Linux is bash -e {0}, which does not set pipefail. The step
pipes go test through tee, so without it the pipeline takes tee's exit
status — always 0 — and the drift job would report success no matter what the test
did.

The drift report matches drift messages, not the marker prefix.
verifyCatalogAlignment emits CATALOG ALIGNMENT: for three outcomes: two that
mean drift (has no conformance test, registers unknown case) and one that does
not (load catalog: read catalog: …, when the clone is missing or the path is
wrong). Grepping the prefix would classify a harness failure as catalog drift,
which is exactly the case the branch exists to separate out.

Scope

No SDK source is touched — five files, all CI. The issuer-identity and PRM
derivation work is deliberately not in here; it lands separately so the mechanical
change and the semantic one can be reviewed on their own terms.

Verification

The fetch script was run end-to-end locally: it fetches and checks out
b4c758a. The conformance suite passes against the catalog it checks out — this
repo already aligns with that revision, so the pin adopts no new cases and
requires no test changes. shellcheck clean; go build, vet, test and
gofmt green across all four modules.

… branch

CI cloned github.com/AuthPlane/conformance at its default branch, so any change
to the catalog reached this repo immediately — a case added there could turn an
unrelated PR red here, with nothing in this repo having changed. Pinning
decouples them: a catalog change arrives only when this repo deliberately bumps
the pin together with the coverage for it.

The revision is single-sourced in a tracked .conformance-catalog-ref, guarded by
a ^[0-9a-f]{40}$ shape check so a branch or tag name cannot silently un-pin CI,
and read by both ci.yml and release.yml through one script rather than a
copy-pasted block — the pin was single-sourced before, the logic reading it was
not.

A weekly conformance-catalog-drift workflow clones the unpinned tip and runs the
alignment assertion, so new cases surface as an early warning instead of a
surprise. Its report step distinguishes real drift from a harness failure by
matching the two messages that mean drift rather than the marker prefix, which is
also emitted when the catalog cannot be read at all. The alignment step declares
shell: bash so the pipeline gets pipefail — without it the step takes tee's exit
status and reports success unconditionally.

The catalog now lands in $RUNNER_TEMP rather than the workspace, so it stays out
of `go list ./...`, coverage globs, and `git add -A` in the release commit.

No SDK source changes. Verified locally: the fetch script runs end-to-end against
the pinned revision, and the conformance suite passes against the catalog it
checks out — the repo already aligns with this revision, so the pin adopts no new
cases.
@RobertoIskandarani
RobertoIskandarani requested a review from a team as a code owner August 7, 2026 16:55
The pin is only useful if a contributor can reproduce CI. CONTRIBUTING and the
conformance-tests README now say where the ref lives, how to check the catalog
out at it, and that adopting new cases means bumping the ref in the same change.

Also drops a clause the pin change made false: the release bot token is no longer
used to check out the conformance catalog — the fetch script clones it directly,
since the repo is public and needs no credential.
@RobertoIskandarani

Copy link
Copy Markdown
Author

Added 35d9f60 — the contributor-facing half of the pin, which I had left out of
the first push.

A pin is only useful if a contributor can reproduce CI. CONTRIBUTING.md and
core/conformancetests/README.md now say where the ref lives, how to check the
catalog out at it, and that adopting new cases means bumping the ref in the same
change — otherwise the first person to run the suite locally gets the tip and a
different result than CI, which is the confusion the pin exists to remove.

It also drops one clause the pin change made false. RELEASE_GUIDE.md said the
release bot token is used "to push the four annotated tags and check out the
conformance catalog
" — the fetch script clones the catalog directly, no
credential involved, since the repo is public. Left the rest of that line alone:
the private repo also changes "four" to "five" there, but that is a module-count
change from unrelated work and does not belong in a pin PR.

Post-create verification re-run: 2 commits, dirty: [].

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant