Add reference-resolution and required-part conformance track (B1b-3b-1) - #102
Merged
Merged
Conversation
Every document-layer conformance fixture built its Dublin Core part with only a `title` term. Both `title` and `creator` are normatively required (Metadata sections 3.3.1 and 3.3.2 — a non-empty string, and at least one value), and dublin-core.schema.json requires both, so the whole document corpus was non-conformant metadata. That is invisible today but blocks the State Machine section 5.4.2 "Missing required metadata — the Dublin Core part or a required term" row, which would otherwise fire on all 51 fixtures the moment it lands. Document fixtures now build their Dublin Core through one dublinCoreJson helper. It serializes with JSON.stringify rather than interpolating the title: a title bearing a quote or backslash would otherwise emit a malformed part that nothing catches, because the reader treats an unloadable Dublin Core as an indeterminate document-ID basis and skips the recompute, while the oracle raises OracleUnsupported and confirm_clean swallows it — a corrupt part would silently buy less verification, not a failure. `creator` is an ARRAY_TERMS member of the metadata projection, so it enters the canonical document ID. The four fixtures carrying a real (non-pending) id are recomputed accordingly; each was re-derived independently and each negative still mismatches for its stated reason. reject-dublincore-duplicate-keys splices its part from the helper so the duplicate key stays the only difference from a conformant one. Container-layer fixtures keep their existing parts: documentVerdict never runs for layer 'container', so their Dublin Core is never read. No reader behaviour changes. check:conformance 274/274, check:fixtures byte-stable, check:document-oracle confirms 47 defects and 5 clean documents, and all 27 gates plus generate:template pass.
Tier 3 part 2 of the document/part layer: the State Machine section 5.4.2 rows for required and path-only-referenced parts, plus reference resolution over content and the out-of-hash annotation layers. Until now a document loaded CLEAN with an absent or unparseable Dublin Core part, a `#`-anchor pointing at nothing, two blocks sharing an id, or a corrupt collaboration file. This closes both fail-opens B1b-3a recorded: the dcResolvable skip, which emitted nothing at all for an unloadable Dublin Core, and the swallowed duplicate-id canonicalization error. The mapper now reports: the Dublin Core part unreferenced, absent, unparseable or missing a required term (Metadata sections 3.3.1/3.3.2); a path-only manifest reference whose target is absent, or present but corrupt; a dangling core anchor, extension-block target field, or extension cross-reference mark; a dangling anchor from an out-of-hash annotation layer; and a duplicate id in the shared identifier namespace. The last is an INTEGRITY-ERROR in every state (Anchors and References section 7.2) — the only such ceiling this layer can assign before B2/B3, because it does not vary by state. A mistyped `metadata` or `metadata.dublinCore` is the manifest missing-or-mistyping-a-required- field REJECT, distinct from genuine absence. Reference resolution lives in a new pure module and resolves against the canonicalizer's own identifier namespace, never a re-derived one: alphaRenameIds' first pass is extracted into an exported collectDefinedIds over a shared walkContentNodes. Walking RAW stored content requires reproducing every canon transform that changes which ids exist — adjacent-text merging, mark dedup within a text node, and derived field deletion, each scoped exactly as canon scopes it — or the resolver disagrees with computeDocumentId in one direction or the other. That agreement is now asserted differentially in test-canonicalize.ts rather than against hand-written expectations. Extension-block interiors are descended here, unlike in the block/mark classifier: the canonicalizer already resolves and relabels anchors inside them, so a check the document ID depends on cannot stop at their boundary. Out-of-hash anchors are resolved only at the positions the extension schemas declare — a generic sweep false-positives, since a phantom carries a full content tree and a collaboration change record carries verbatim block snapshots naming removed content. Adds ten errors.json codes (vocabulary v0.8), 21 fixtures and independent oracle confirmers. Deferred with recorded rationale: the asset and presentation rows, the hash-bound missing-part row, and the citation/glossary cross-reference namespaces (B1b-3b-2); content-schema validity and NFC (B1b-3c); every frozen/published escalation (B3). check:conformance 199 vectors + 98 fixtures; document oracle confirms 64 injected defects and 11 clean documents; all 29 gates green. Note: inMarks now requires Array.isArray, matching rewriteIds' dispatch. This changes computeDocumentId output for a non-array `marks`, which is schema-invalid content only, but it is a change to the hashing definition and is called out here deliberately.
6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Tier 3 part 2 of the document/part layer: the State Machine §5.4.2 rows for required and path-only-referenced parts, plus reference resolution over content and the out-of-hash annotation layers.
Until now a document loaded CLEAN with an absent or unparseable Dublin Core part, a
#-anchor pointing at nothing, two blocks sharing an id, or a corrupt collaboration file. This closes both fail-opens B1b-3a recorded: thedcResolvableskip, which emitted nothing at all for an unloadable Dublin Core, and the swallowed duplicate-id canonicalization error.Rows implemented (all draft/review; every frozen/published escalation stays in B3):
The id collision is the only INTEGRITY-ERROR this layer can assign before B2/B3, precisely because it does not vary by state. A mistyped
metadataormetadata.dublinCoreis the manifest missing-or-mistyping-a-required-field REJECT, distinct from genuine absence.Reference resolution lives in a new pure module and resolves against the canonicalizer's own identifier namespace, never a re-derived one:
alphaRenameIds' first pass is extracted into an exportedcollectDefinedIdsover a sharedwalkContentNodes. Walking RAW stored content requires reproducing everycanontransform that changes which ids exist — adjacent-text merging, mark dedup within a text node, and derived-field deletion, each scoped exactly ascanonscopes it — or the resolver disagrees withcomputeDocumentIdin one direction or the other. That agreement is asserted differentially intest-canonicalize.tsrather than against hand-written expectations.Extension-block interiors are descended here, unlike in the block/mark classifier: the canonicalizer already resolves and relabels anchors inside them, so a check the document ID depends on cannot stop at their boundary. Out-of-hash anchors are resolved only at the positions the extension schemas declare — a generic sweep false-positives, since a phantom carries a full content tree and a collaboration change record carries verbatim block snapshots naming removed content.
Adds 10
errors.jsoncodes (vocabulary v0.8), 21 fixtures, and independent oracle confirmers.Three things worth your attention
1. A change to the hashing definition.
inMarksnow requiresArray.isArray, matchingrewriteIds' dispatch. This alterscomputeDocumentIdoutput for a non-arraymarks(sha256:e6d63020…→sha256:60ed760a…). Only schema-invalid content is affected —content.schema.jsontypesmarksas an array — and the new behaviour is the correct one, but this repo has a no-signed-byte-drift discipline and the change rode in on a branch named for the reference resolver. Flagging it rather than letting it pass silently. Say if you want adocument-idvector pinning it.2. The oracle is a transliteration for this slice, not an independent implementation. Its ZIP parsing, duplicate-key scanner and
confirm_cleanaggregates remain genuinely independent and catch a fixture whose bytes do not carry the defect itscase.jsonclaims. But_walk_content/_merge_key/_collect_idsare ports of the TypeScript, and during review they reproduced two critical defects verbatim rather than catching them. For the raw-walk and merge logic the real oracle is the differential test againstcomputeDocumentId. The "64 defects confirmed" figure should be read with that in mind.3. Three spec questions, recorded rather than guessed at:
metadata.dublinCoreis a required manifest field, so a manifest omittingmetadataentirely also reads as the "missing a required field" REJECT. I took the more specific metadata row (WARNING) and authoredwarn-metadata-unreferencedas an interval[WARNING, REJECT]so the suite does not foreclose either reading. Your call whether to settle it.footnotemark's target namespace. The mark carries an optionalid, butsemantic:footnoteblocks match bynumber/symbol, soidaddresses nothing clearly specified. Deferred outright.semantic.schema.json. The row calls the bibliography/glossary side files "path-only", but the schema declares them{path, hash}, which the projector binds — making them row 310, not 318. Flagged rather than propagated into a published note.Test plan
check:conformance— 199 vectors + 98 fixturescheck:document-oracle— 64 injected defects confirmed, 11 clean documentstest:canonicalize— 96/96, including the differential raw-walk-vs-canontestcheck:document-id,check:canonicalize-oracle,check:manifest-projection— canonicalize-refactor regressionscheck:fixturesbyte round-trip;check:enumeration-coverage;generate:templateReview history
Three
/review-fixcycles across five independent reviewers found 4 CRITICAL defects that all green gates missed — three of them false or missed INTEGRITY-ERRORs caused by the raw walk diverging fromcanon, and one duplicate-key REJECT lost for parts not named*.json. Two were introduced by earlier fixes in the same run. All are fixed and mutation-verified.