Skip to content

Add reference-resolution and required-part conformance track (B1b-3b-1) - #102

Merged
gvonnessi merged 2 commits into
mainfrom
feat/conformance-b1b3b1-reference-resolver
Jul 28, 2026
Merged

Add reference-resolution and required-part conformance track (B1b-3b-1)#102
gvonnessi merged 2 commits into
mainfrom
feat/conformance-b1b3b1-reference-resolver

Conversation

@gvonnessi

Copy link
Copy Markdown
Collaborator

Summary

Tier 3 part 2 of the document/part layer: the State Machine §5.4.2 rows for required and path-only-referenced parts, plus reference resolution over content and the out-of-hash annotation layers.

Until now a document loaded CLEAN with an absent or unparseable Dublin Core part, a #-anchor pointing at nothing, two blocks sharing an id, or a corrupt collaboration file. This closes both fail-opens B1b-3a recorded: the dcResolvable skip, which emitted nothing at all for an unloadable Dublin Core, and the swallowed duplicate-id canonicalization error.

Rows implemented (all draft/review; every frozen/published escalation stays in B3):

Row Check Disposition
309 Dublin Core unreferenced / absent / unparseable / missing a required term WARNING
311 A path-only manifest reference whose target is absent WARNING (state-invariant)
313 Dangling core anchor; dangling extension-block target field (separate code) WARNING
316 Dangling extension cross-reference mark WARNING (state-invariant)
317 Dangling anchor from an out-of-hash annotation layer WARNING (state-invariant)
318 Out-of-hash extension data part present but unparseable WARNING (state-invariant)
03a §7.2 Duplicate id in the shared identifier namespace INTEGRITY-ERROR, every state

The id collision is the only INTEGRITY-ERROR this layer can assign before B2/B3, precisely because it does not vary by state. A mistyped metadata or metadata.dublinCore is the manifest missing-or-mistyping-a-required-field REJECT, distinct from genuine absence.

Reference resolution lives in a new pure module and resolves against the canonicalizer's own identifier namespace, never a re-derived one: alphaRenameIds' first pass is extracted into an exported collectDefinedIds over a shared walkContentNodes. Walking RAW stored content requires reproducing every canon transform that changes which ids exist — adjacent-text merging, mark dedup within a text node, and derived-field deletion, each scoped exactly as canon scopes it — or the resolver disagrees with computeDocumentId in one direction or the other. That agreement is asserted differentially in test-canonicalize.ts rather than against hand-written expectations.

Extension-block interiors are descended here, unlike in the block/mark classifier: the canonicalizer already resolves and relabels anchors inside them, so a check the document ID depends on cannot stop at their boundary. Out-of-hash anchors are resolved only at the positions the extension schemas declare — a generic sweep false-positives, since a phantom carries a full content tree and a collaboration change record carries verbatim block snapshots naming removed content.

Adds 10 errors.json codes (vocabulary v0.8), 21 fixtures, and independent oracle confirmers.

Three things worth your attention

1. A change to the hashing definition. inMarks now requires Array.isArray, matching rewriteIds' dispatch. This alters computeDocumentId output for a non-array marks (sha256:e6d63020…sha256:60ed760a…). Only schema-invalid content is affected — content.schema.json types marks as an array — and the new behaviour is the correct one, but this repo has a no-signed-byte-drift discipline and the change rode in on a branch named for the reference resolver. Flagging it rather than letting it pass silently. Say if you want a document-id vector pinning it.

2. The oracle is a transliteration for this slice, not an independent implementation. Its ZIP parsing, duplicate-key scanner and confirm_clean aggregates remain genuinely independent and catch a fixture whose bytes do not carry the defect its case.json claims. But _walk_content / _merge_key / _collect_ids are ports of the TypeScript, and during review they reproduced two critical defects verbatim rather than catching them. For the raw-walk and merge logic the real oracle is the differential test against computeDocumentId. The "64 defects confirmed" figure should be read with that in mind.

3. Three spec questions, recorded rather than guessed at:

  • §5.4.2 row 299 vs row 309. metadata.dublinCore is a required manifest field, so a manifest omitting metadata entirely also reads as the "missing a required field" REJECT. I took the more specific metadata row (WARNING) and authored warn-metadata-unreferenced as an interval [WARNING, REJECT] so the suite does not foreclose either reading. Your call whether to settle it.
  • The footnote mark's target namespace. The mark carries an optional id, but semantic:footnote blocks match by number/symbol, so id addresses nothing clearly specified. Deferred outright.
  • §5.4.2 row 318 vs semantic.schema.json. The row calls the bibliography/glossary side files "path-only", but the schema declares them {path, hash}, which the projector binds — making them row 310, not 318. Flagged rather than propagated into a published note.

Test plan

  • check:conformance — 199 vectors + 98 fixtures
  • check:document-oracle — 64 injected defects confirmed, 11 clean documents
  • test:canonicalize — 96/96, including the differential raw-walk-vs-canon test
  • check:document-id, check:canonicalize-oracle, check:manifest-projection — canonicalize-refactor regressions
  • check:fixtures byte round-trip; check:enumeration-coverage; generate:template
  • All 29 gates green locally
  • Every new defense mutation-tested: disabling any check, erasure, or reference field fails the gate; both positive controls fail when flipped

Review history

Three /review-fix cycles across five independent reviewers found 4 CRITICAL defects that all green gates missed — three of them false or missed INTEGRITY-ERRORs caused by the raw walk diverging from canon, and one duplicate-key REJECT lost for parts not named *.json. Two were introduced by earlier fixes in the same run. All are fixed and mutation-verified.

Every document-layer conformance fixture built its Dublin Core part with
only a `title` term. Both `title` and `creator` are normatively required
(Metadata sections 3.3.1 and 3.3.2 — a non-empty string, and at least one
value), and dublin-core.schema.json requires both, so the whole document
corpus was non-conformant metadata. That is invisible today but blocks the
State Machine section 5.4.2 "Missing required metadata — the Dublin Core
part or a required term" row, which would otherwise fire on all 51
fixtures the moment it lands.

Document fixtures now build their Dublin Core through one dublinCoreJson
helper. It serializes with JSON.stringify rather than interpolating the
title: a title bearing a quote or backslash would otherwise emit a
malformed part that nothing catches, because the reader treats an
unloadable Dublin Core as an indeterminate document-ID basis and skips the
recompute, while the oracle raises OracleUnsupported and confirm_clean
swallows it — a corrupt part would silently buy less verification, not a
failure.

`creator` is an ARRAY_TERMS member of the metadata projection, so it
enters the canonical document ID. The four fixtures carrying a real
(non-pending) id are recomputed accordingly; each was re-derived
independently and each negative still mismatches for its stated reason.
reject-dublincore-duplicate-keys splices its part from the helper so the
duplicate key stays the only difference from a conformant one.

Container-layer fixtures keep their existing parts: documentVerdict never
runs for layer 'container', so their Dublin Core is never read.

No reader behaviour changes. check:conformance 274/274, check:fixtures
byte-stable, check:document-oracle confirms 47 defects and 5 clean
documents, and all 27 gates plus generate:template pass.
Tier 3 part 2 of the document/part layer: the State Machine section 5.4.2
rows for required and path-only-referenced parts, plus reference
resolution over content and the out-of-hash annotation layers. Until now
a document loaded CLEAN with an absent or unparseable Dublin Core part, a
`#`-anchor pointing at nothing, two blocks sharing an id, or a corrupt
collaboration file. This closes both fail-opens B1b-3a recorded: the
dcResolvable skip, which emitted nothing at all for an unloadable Dublin
Core, and the swallowed duplicate-id canonicalization error.

The mapper now reports: the Dublin Core part unreferenced, absent,
unparseable or missing a required term (Metadata sections 3.3.1/3.3.2); a
path-only manifest reference whose target is absent, or present but
corrupt; a dangling core anchor, extension-block target field, or
extension cross-reference mark; a dangling anchor from an out-of-hash
annotation layer; and a duplicate id in the shared identifier namespace.
The last is an INTEGRITY-ERROR in every state (Anchors and References
section 7.2) — the only such ceiling this layer can assign before B2/B3,
because it does not vary by state. A mistyped `metadata` or
`metadata.dublinCore` is the manifest missing-or-mistyping-a-required-
field REJECT, distinct from genuine absence.

Reference resolution lives in a new pure module and resolves against the
canonicalizer's own identifier namespace, never a re-derived one:
alphaRenameIds' first pass is extracted into an exported
collectDefinedIds over a shared walkContentNodes. Walking RAW stored
content requires reproducing every canon transform that changes which ids
exist — adjacent-text merging, mark dedup within a text node, and derived
field deletion, each scoped exactly as canon scopes it — or the resolver
disagrees with computeDocumentId in one direction or the other. That
agreement is now asserted differentially in test-canonicalize.ts rather
than against hand-written expectations.

Extension-block interiors are descended here, unlike in the block/mark
classifier: the canonicalizer already resolves and relabels anchors
inside them, so a check the document ID depends on cannot stop at their
boundary. Out-of-hash anchors are resolved only at the positions the
extension schemas declare — a generic sweep false-positives, since a
phantom carries a full content tree and a collaboration change record
carries verbatim block snapshots naming removed content.

Adds ten errors.json codes (vocabulary v0.8), 21 fixtures and independent
oracle confirmers. Deferred with recorded rationale: the asset and
presentation rows, the hash-bound missing-part row, and the
citation/glossary cross-reference namespaces (B1b-3b-2); content-schema
validity and NFC (B1b-3c); every frozen/published escalation (B3).

check:conformance 199 vectors + 98 fixtures; document oracle confirms 64
injected defects and 11 clean documents; all 29 gates green.

Note: inMarks now requires Array.isArray, matching rewriteIds' dispatch.
This changes computeDocumentId output for a non-array `marks`, which is
schema-invalid content only, but it is a change to the hashing definition
and is called out here deliberately.
@gvonnessi
gvonnessi merged commit fcdf4da into main Jul 28, 2026
2 checks passed
@gvonnessi
gvonnessi deleted the feat/conformance-b1b3b1-reference-resolver branch July 28, 2026 10:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant