Skip to content

feat: nightly schema audit against the live API - #154

Merged
MrRefactoring merged 2 commits into
developfrom
feature/schema-audit
Jul 20, 2026
Merged

feat: nightly schema audit against the live API#154
MrRefactoring merged 2 commits into
developfrom
feature/schema-audit

Conversation

@MrRefactoring

Copy link
Copy Markdown
Owner

Adds a nightly run that asks what the existing live suite cannot: not
"does the library still work", but "do the schemas still describe what
Confluence actually sends".

Loose validation cannot answer that by construction — letting
undocumented keys through is exactly what it is for — so the audit flips
every response schema to strict.

Nothing changes for callers. Validation stays loose, undocumented
fields keep passing through, and the published declarations are
byte-identical (verified before and after: the return type of
apiObject is still $loose). The audit is off unless
AUDIT_SCHEMAS=true, which only this workflow sets.

First run

161 undocumented fields across 93 endpoints, with the suite passing
493/493. Heavily concentrated, so this is roughly a dozen fixes rather
than 161:

Field Endpoints
_links.base 77
_links.context 53
_links.self 39
version.ncsStepVersion 18
_links.edituiv2 16

Why the findings are recorded rather than thrown

The obvious implementation — make schemas strict and let them fail — was
tried first and does not work here. Three things came out of running it:

  • One stale schema ended the run. A single extra key in the space
    fixture threw in beforeAll and skipped all 19 tests in the file.
    Findings are now recorded and the response re-parsed without the
    offending keys, so the whole suite is always covered.
  • Returning the raw body broke the transforms. z.coerce.date()
    only runs on a successful parse, so every Date came back a string
    and tests failed for reasons that said nothing about drift.
  • Union responses were a blind spot. Zod wraps branch issues in
    invalid_union, so a flat check misses them entirely. Handling them
    surfaced 41 further fields across 5 endpoints that the report had
    been silently missing — a report that looks complete without being so
    is worse than no report.

Anything that is not an undocumented key still throws. A missing field
or a changed type is real breakage and must not be absorbed.

Why a separate workflow

live-tests.yml already runs nightly, and folding this into it was
tempting. The two report different things: a field that vanished breaks
consumers, a field that appeared is documentation debt. Mixed together,
the first would be buried under a pile of the second.

Runs at 04:00 UTC, an hour behind the live suite, in the same
live-tests concurrency group — one tenant serves both.

Notes

  • The run is red until the drift is worked through. Failing on any
    drift was a deliberate choice over baselining the known 161.
  • Findings are kept in memory rather than written from core/, which
    ships to browsers and cannot reach the filesystem; the test-side
    collector writes them out.
  • The report collapses array indices and resource ids, so
    results.0._links.self and /attachments/att17105305 do not fill it
    with the same finding under ids that lived for one run.
  • This measures a lower bound: only what this tenant returns. Fields
    that appear under other features or plans will not show up.

Adds `audit:schemas` and a nightly workflow that asks a question the
existing live run cannot: not "does the library still work", but "do the
schemas still describe what Confluence actually sends".

Kept as a separate workflow rather than folded into live-tests, because
the two report different things. A field that vanished breaks consumers;
a field that appeared is documentation debt. Mixing them would bury the
first under a pile of the second, and a nightly run nobody reads is worth
nothing.

The report aggregates by field rather than by occurrence, and collapses
array indices and resource ids — `results.0._links.self` and
`results.1._links.self` are one gap in one schema, and
`/attachments/att17105305` names an id that existed for the length of one
run. Without that the summary lists the same finding dozens of times and
cannot be compared between runs.

First run: 161 undocumented fields across 93 endpoints, concentrated —
`_links.base` alone accounts for 77 of them, and thirteen fields cover
most of the rest. The suite itself passes 493/493, so a red run here now
means either real breakage or new drift, and the summary says which.
@MrRefactoring
MrRefactoring merged commit 01eebdd into develop Jul 20, 2026
7 checks passed
@MrRefactoring
MrRefactoring deleted the feature/schema-audit branch July 20, 2026 15:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant