Skip to content

feat(conformance): CTK vectors for the §12.1 incremental exception - #67

Merged
MohammadHaroonAbuomar merged 1 commit into
mainfrom
mhabuomar/incremental-conformance-vectors
Aug 7, 2026
Merged

feat(conformance): CTK vectors for the §12.1 incremental exception#67
MohammadHaroonAbuomar merged 1 commit into
mainfrom
mhabuomar/incremental-conformance-vectors

Conversation

@MohammadHaroonAbuomar

Copy link
Copy Markdown
Contributor

Implements R7 from the streaming comparison study (agent-control-spec analysis/streaming-acs-vs-maf.md): the conformance vectors #64 promised as future work, making buffered_output: false a CTK-testable surface instead of declaration-only. Additive; spec version stays agent-hooks/0.1 (0.1.0-alpha line).

The vector part

New streaming/incremental part, one vector per §12.1 exception condition:

Vector Pins
AH-CTK-110 release segment by segment under covering verdicts (item 1)
AH-CTK-111 terminating deny withholds the unreleased remainder, harmless undelivered tail included; released prefix stays released (items 1–2)
AH-CTK-112 residue no emission evaluated fails closed with host_error:streaming_unsupported (item 3)
AH-CTK-113 durability gated by the same discipline as release; withheld/unevaluated content never durable (item 4, §6.1)

Grammar and gating

  • respond.stream / respond.stream_truncated: a chunked mock stream; chunk boundaries pin the evaluated segments (prefix emissions, finish_reason: "incremental" for non-final segments), and truncation pins the residue shape.
  • expect.released_output / expect.persisted_must_not_contain: the egress and durability observables, backed by two RunRecord fields the declaring harness reports.
  • New incremental_output capability gates the part via the existing subset-skip mechanism: a host declaring buffered_output: true (the default) never declares it and skips. The vectors pin the watermark-gated posture (declared exposure bound: none — ACS §18.1 blocking); a looser-bound host also skips, documented in HARNESS.md.
  • Reference-harness skip manifests in all five SDKs grow by the four IDs (no in-tree reference declares the capability — every reference harness buffers).

Honest limits, stated in the artefacts

  • Runner-side assertion support for released_output/persisted_must_not_contain lands with the first declaring host (HARNESS.md says so); today the part is inert-by-gating for every surface, so nothing dead ships in the engines.
  • With the watermark-gated bound, item 2's withheld-but-permitted clause is structurally empty on the egress side (a permitted segment is released on its verdict); AH-CTK-111's title says so and AH-CTK-113 exercises the clause on the durable side.
  • A host that persists nothing passes the durability assertions vacuously — which is exactly the always-safe §6.1 posture, stated in the schema description.

Docs updated (the #64 future-work notes)

Spec §12.1 (future-work note → pointer to the part), §13.1, §13.3, conformance/CLAIMS.md, conformance/HARNESS.md (drive conventions + coverage boundaries), docs/PRODUCTION.md row 8, docs/THREAT-MODEL.md TM-11 (gap narrowed to the buffered surface), CHANGELOG.

Verification

  • ajv validate over the corpus with the amended schema: all 51 valid; schema compiles.
  • Rust CTK reference self-test green with the pinned skip set.
  • ACS cross-check: agent-control-spec's conformance suite (its Rust harness at current main) run with the vendored corpus swapped for this branch's: 46 passed, 0 failed, 5 skipped of 51 (previously 46/47 with 1 skip). The four new skips are the capability gate doing its job; the 46-pass surface is intact.

@MohammadHaroonAbuomar
MohammadHaroonAbuomar force-pushed the mhabuomar/incremental-conformance-vectors branch from 1f4482c to 6c94017 Compare August 7, 2026 06:23
@MohammadHaroonAbuomar MohammadHaroonAbuomar added the spec:additive Additive normative change (optional fields, new vectors) label Aug 7, 2026
…tion

Deliver the vectors PR #64 left as future work, making the
buffered_output: false posture CTK-testable (streaming comparison
study, agent-control-spec analysis/streaming-acs-vs-maf.md, R7).

New streaming/incremental vector part, one vector per exception
condition:

- AH-CTK-110: a fully permitted stream is released segment by
  segment under covering verdicts (item 1).
- AH-CTK-111: a terminating deny withholds the unreleased remainder,
  the harmless undelivered tail included, while the released prefix
  stays released (items 1-2).
- AH-CTK-112: residue no emission evaluated fails closed at end of
  stream with host_error:streaming_unsupported (item 3), via the new
  stream_truncated scenario form.
- AH-CTK-113: durable incorporation is gated by the same discipline
  as release; withheld and unevaluated content never becomes durable
  (item 4, section 6.1).

The vector grammar grows respond.stream / respond.stream_truncated
(chunked mock stream; chunk boundaries pin the evaluated segments)
and expect.released_output / expect.persisted_must_not_contain (the
egress and durability observables). The part is gated on the new
incremental_output capability: buffering hosts (buffered_output:
true, the default) do not declare it and skip, so the vectors are
additive and no existing declared surface changes. Reference-harness
skip manifests in all five SDKs pin the new skips; the Python
Capability enum and vendored vector copies are updated.

Spec section 12.1's future-work note now points at the part; section
13.1/13.3, CLAIMS.md, HARNESS.md (incremental drive conventions and
coverage boundaries), PRODUCTION.md row 8, and THREAT-MODEL.md TM-11
are updated to match. Spec version unchanged (agent-hooks/0.1,
0.1.0-alpha).

Verified: ajv vector validation green; Rust CTK reference self-test
green with the pinned skip set; agent-control-spec's conformance
suite run against this corpus reports 46 passed, 0 failed, 5 skipped
of 51 (the four new skips are the capability gate working).

Signed-off-by: MohammadHaroonAbuomar <40180927+MohammadHaroonAbuomar@users.noreply.github.com>
@MohammadHaroonAbuomar
MohammadHaroonAbuomar force-pushed the mhabuomar/incremental-conformance-vectors branch from 6c94017 to 0051e03 Compare August 7, 2026 06:24
@MohammadHaroonAbuomar
MohammadHaroonAbuomar merged commit 473af01 into main Aug 7, 2026
29 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

spec:additive Additive normative change (optional fields, new vectors)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant