feat: nightly schema audit against the live API - #154
Merged
Conversation
Adds `audit:schemas` and a nightly workflow that asks a question the existing live run cannot: not "does the library still work", but "do the schemas still describe what Confluence actually sends". Kept as a separate workflow rather than folded into live-tests, because the two report different things. A field that vanished breaks consumers; a field that appeared is documentation debt. Mixing them would bury the first under a pile of the second, and a nightly run nobody reads is worth nothing. The report aggregates by field rather than by occurrence, and collapses array indices and resource ids — `results.0._links.self` and `results.1._links.self` are one gap in one schema, and `/attachments/att17105305` names an id that existed for the length of one run. Without that the summary lists the same finding dozens of times and cannot be compared between runs. First run: 161 undocumented fields across 93 endpoints, concentrated — `_links.base` alone accounts for 77 of them, and thirteen fields cover most of the rest. The suite itself passes 493/493, so a red run here now means either real breakage or new drift, and the summary says which.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a nightly run that asks what the existing live suite cannot: not
"does the library still work", but "do the schemas still describe what
Confluence actually sends".
Loose validation cannot answer that by construction — letting
undocumented keys through is exactly what it is for — so the audit flips
every response schema to strict.
Nothing changes for callers. Validation stays loose, undocumented
fields keep passing through, and the published declarations are
byte-identical (verified before and after: the return type of
apiObjectis still$loose). The audit is off unlessAUDIT_SCHEMAS=true, which only this workflow sets.First run
161 undocumented fields across 93 endpoints, with the suite passing
493/493. Heavily concentrated, so this is roughly a dozen fixes rather
than 161:
_links.base_links.context_links.selfversion.ncsStepVersion_links.edituiv2Why the findings are recorded rather than thrown
The obvious implementation — make schemas strict and let them fail — was
tried first and does not work here. Three things came out of running it:
fixture threw in
beforeAlland skipped all 19 tests in the file.Findings are now recorded and the response re-parsed without the
offending keys, so the whole suite is always covered.
z.coerce.date()only runs on a successful parse, so every
Datecame back a stringand tests failed for reasons that said nothing about drift.
invalid_union, so a flat check misses them entirely. Handling themsurfaced 41 further fields across 5 endpoints that the report had
been silently missing — a report that looks complete without being so
is worse than no report.
Anything that is not an undocumented key still throws. A missing field
or a changed type is real breakage and must not be absorbed.
Why a separate workflow
live-tests.ymlalready runs nightly, and folding this into it wastempting. The two report different things: a field that vanished breaks
consumers, a field that appeared is documentation debt. Mixed together,
the first would be buried under a pile of the second.
Runs at 04:00 UTC, an hour behind the live suite, in the same
live-testsconcurrency group — one tenant serves both.Notes
drift was a deliberate choice over baselining the known 161.
core/, whichships to browsers and cannot reach the filesystem; the test-side
collector writes them out.
results.0._links.selfand/attachments/att17105305do not fill itwith the same finding under ids that lived for one run.
that appear under other features or plans will not show up.