v1 identity, spec corrections, and conformance-suite hardening - #40
Open
jalexspringer wants to merge 8 commits into
Open
v1 identity, spec corrections, and conformance-suite hardening#40jalexspringer wants to merge 8 commits into
jalexspringer wants to merge 8 commits into
Conversation
The spec body already makes v1 normative claims (ip_hash withdrawal in 9.1, the content_displayed rename in 12.1) while the header still said 0.1/Preview and every schema pinned schema_version to "0.1" under a /schema/v0.1/ $id, so a conforming v1 emitter had to declare "0.1" and the two versions were indistinguishable on the wire. - Header: Version 1.0 (release candidate draft), status release candidate in preparation, feature freeze 21 August 2026. - All four schemas: $id path v0.1 -> v1, schema_version const -> "1.0"; manifest const description now says v1 emitters MUST use "1.0". - 5.7.4 and 12.1 now state explicitly that v1 documents declare "1.0", that a v0.1 consumer rejects them under the preview rule, and that a v1 consumer rejects "0.1". - Every fixture and every inline example in SPECIFICATION.md and README.md now declares "1.0". invalid/invalid-schema-version.json keeps its deliberately wrong "2.0" (still != the new const). - validate.py / tests/README.md docstrings updated; no hardcoded versions or v0.1 paths remain in the test scripts. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The table said id was optional and generated by the server if not provided. The schema requires id on content_cited, content_reproduced and content_presented, and server-side generation is incompatible with citation_id and presentation_id, which reference an id the emitter must already hold when it constructs the referencing event. Reworded to the table's conditional style (as used by output_id and presentation_id): required for reproduced/cited/presented, optional elsewhere, emitter-assigned. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Section 5.7.3 lists data.citation_type among the Citation-emitter requirements marked schema-enforced, but the content_cited conditional required only id and output_id - unlike content_reproduced and content_presented, whose conditionals require their data objects. The content_cited conditional now requires data with citation_type, mirroring the reproduced (data.reproduction_type) and presented (data.presentation_kind/presentation_type) pattern. Two cited events carried no data and gained citation_type: "reference" (each is paired with a link presentation of the same source): the 7.1 event-batch example and tests/valid/event-batch-agent.json, plus the same event in tests/invalid/batch-missing-session-and-ctx-token.json so that fixture still passes the schema and fails at the application layer as its description intends. All other cited fixtures and examples already carried citation_type. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- 8.3: revert 'Presentation name' to 'Display name' for operator.name. The display->presentation rename applies to the presentation event, not ordinary UI terminology; manifest.json already says Display name. - telemetry-session.json: ad_rendered description now says 'rendered', matching the field name and 5.4. - 6.5: restate the excerpt pair on the v1 chars-primary hierarchy - excerpt_chars is the portable primary measurement under 6.4's counting rule, excerpt_tokens the agent-native supplementary one - mirroring 6.4's chars_ingested/tokens_ingested wording so 6.6's 'same pairing' cross-reference holds. - 6 intro: the profiles are no longer in lifecycle order (6.5 Citation precedes 6.6 Reproduction while the lifecycle runs Reproduced then Cited), so the intro no longer claims they are; sections keep their numbers. - 5.7: the optional-signals sentence now names reproduction alongside presentation and engagement - it likewise sits outside the Retrieval/Grounding/Citation ladder as a SHOULD. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…shapes Mutation-verified review findings, all previously undetected: - Build every validator with format_checker so format: uuid / date-time / uri assertions enforce instead of annotate; both runners hard-error at startup if the checker lacks uuid or date-time. Install line becomes pip install "jsonschema[format-nongpl]". - Require _expected_error on every invalid fixture: a substring that must appear in the actual error (first schema error message and JSON pointer, or the application-layer violation text). A fixture that fails for the wrong reason, or carries no pin, now fails the run. - Reconcile APPLICATION_LAYER_VIOLATIONS keys against invalid/: an entry with no matching file fails the run instead of silently dropping the expectation. - Apply privacy field gating (5.5) to turns wherever they appear: session documents, event batches, and standalone event envelopes, not only session events lists. - Add referential integrity checks within a session document (6.6-6.8): content_engaged.presentation_id must match a content_presented event id, and citation_id on content_presented/content_reproduced must match a content_cited event id. Standalone envelopes and batch members are exempt; the corroborating click-out flow is out of scope here. - check_examples.py: validate complete bare event objects (type + timestamp) against the TelemetryEvent definition instead of skipping them; 3 of the 7 skipped fragments are now validated. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
New invalid fixtures, each pinned by _expected_error:
- presented-missing-id, presented-missing-output-id, cited-missing-id:
the required event and output identifiers of sections 6.5 and 6.7.
- cited-missing-citation-type: the schema now requires
data.citation_type on content_cited.
- reproduction-type-invalid: reproduction_type 'paraphrase' - section
6.6 says paraphrase is not reproduction.
- presentation-kind-invalid: presentation_kind is a closed two-value
enum.
- grounded-negative-chars-ingested, reproduced-negative-chars: count
fields carry minimum 0.
- malformed-parent-session-id: format: uuid, caught only now that the
runners enforce format assertions.
- privacy-violation-{response-text,query-intent,topics,response-type,
response-mode,model-id}-at-minimal: one fixture per remaining field
forbidden at minimal privacy (section 5.5).
- presented/retrieved/engaged-missing-identifier: the section 5.7.5
identifier rule, previously only tested via content_grounded.
- engaged-presentation-id-unmatched, presented-citation-id-unmatched:
the new intra-session referential integrity checks (sections 6.7,
6.8).
New valid fixture:
- session-reproduction-no-grounding: the fifth funnel departure of
section 4.3, reproduction of memorised content with no grounding
event.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
mutation_smoke.py copies the schemas and tests/ into a temp directory, applies each known suite-weakening mutation - dropping the format checker, gutting the withdrawn-ip-hash fixture, shrinking CONTENT_EVENT_TYPES and PRIVACY_FORBIDDEN_FIELDS, pointing an engagement at an all-zeros presentation_id - and confirms validate.py fails under every one. Each of these previously went undetected. The working tree is never modified; a surviving mutation fails the script. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Review fixes from the 12 Aug v1-draft review, in three groups.
v1 version identity
The spec header still declared Version 0.1 / Preview while the body made v1 normative claims (§9.1 ip_hash withdrawal, §12.1 event rename), and the schemas kept
/schema/v0.1/$ids withschema_versionconst"0.1"— so the "v0.1" schemas rejected valid v0.1 documents, a conforming v1 emitter had to declare0.1, and v1/v0.1 were indistinguishable on the wire. Now: header is 1.0 (release candidate draft) with the 21 Aug freeze noted, schemas declare/schema/v1/andschema_version "1.0", §5.7.4/§12.1 state the negotiation explicitly, and every fixture and inline example is swept (invalid-schema-version.jsondeliberately keeps a wrong version).Spec/schema corrections
idrow said optional, "generated by server if not provided" — contradicting the schema (required on reproduced/cited/presented) and incompatible withcitation_id/presentation_idreferences. Now conditional and emitter-assigned.data.citation_typeis schema-enforced; the schema didn't require it. Thecontent_citedconditional now requiresdata.citation_type, mirroring reproduced/presented.ad_rendereddescription aligned to "rendered"; §6.5 updated to the chars-primary/tokens-supplementary hierarchy that §6.6 already cites; §6 intro no longer claims lifecycle ordering; §5.7 overview now names reproduction among the optional lifecycle signals.Conformance-suite hardening (review findings were mutation-verified)
"not-a-uuid"session ids validated) plus a startup guard; install line ispip install "jsonschema[format-nongpl]"._expected_error; a fixture failing for the wrong reason fails the run (previously, gutting a fixture left the suite green). OrphanedAPPLICATION_LAYER_VIOLATIONSkeys fail the run.content_engaged.presentation_idmust match acontent_presented.id,citation_idmust match acontent_cited.id.tests/mutation_smoke.pyreplays the review's five suite-weakening mutations; all now caught.Explicitly out of scope (owned by
v1-click-context)§7.1's click-manifest event set and the
content_engagedpresentation_idconditionality. Two spec ambiguities surfaced while writing fixtures and are left open for that work: neither §6.7 nor §6.8 states the document scope ofcitation_id/presentation_idreferences (the suite enforces them within session documents only; standalone/batch envelopes are exempt — mixed per-session delivery is undefined in the spec).Remaining editorial follow-up (deliberately untouched): prose that discusses v0.1 as a historical version (§8.4 signing "informational in v0.1", §12.1 migration narrative, README "Current spec version: 0.1") needs an editorial pass deciding which policies carry into v1.
🤖 Generated with Claude Code