feat(conformance): CTK vectors for the §12.1 incremental exception

CHANGELOG.md

@@ -3,6 +3,23 @@
 User-visible changes to the spec and SDKs. Versioning rules:
 [VERSIONING.md](VERSIONING.md).
 
+## Unreleased
+
+- **The §12.1 incremental exception is CTK-testable.** The vectors
+  the alpha.5 entry below left as future work exist: a
+  `streaming/incremental` part (`AH-CTK-110`–`AH-CTK-113`) exercises
+  the exception's four conditions against a chunked mock stream —
+  release under covering verdicts, a terminating deny that withholds
+  the unreleased remainder, uncleared residue failing closed with
+  `host_error:streaming_unsupported`, and §6.1-gated durability. The
+  vector grammar grows `respond.stream`/`respond.stream_truncated` and
+  `expect.released_output`/`expect.persisted_must_not_contain`; the
+  part is gated on the new `incremental_output` capability, so every
+  buffering host (`buffered_output: true`, the default) skips it —
+  the vectors are additive and no existing declared surface changes.
+  Reference-harness skip manifests across the five SDKs pin the new
+  skips. Spec version unchanged (`agent-hooks/0.1`, 0.1.0-alpha).
+
 ## 0.1.0-alpha.5 — tag `v0.1.0-alpha.5`
 
 - **Python: `Verdict.allow()` constructor sugar,** completing the

conformance/CLAIMS.md

@@ -25,7 +25,10 @@ property (§10.1). A claim with
 `buffered_output: false` MUST state that a `deny` at `output` cannot
 retract already-streamed content (§12.1a); one whose host mediates
 incrementally under the §12.1 exception MUST also state the exposure
-bound its accounting discipline enforces.
+bound its accounting discipline enforces and MUST declare
+`incremental_output`, so the `streaming/incremental` vectors
+(`AH-CTK-110`–`AH-CTK-113`) run against that discipline instead of
+being skipped.
 
 | Framework | Adapter version | Spec | Capabilities | Profiles | Identity provider | SDK | Report | Notes |
 | --- | --- | --- | --- | --- | --- | --- | --- | --- |
@@ -50,8 +53,10 @@ artefacts, in the PR:
    the claim states records/approvals are identity-unbound;
    custom provider → content-derived or not; `buffered_output: false`
    → the claim states a deny at `output` cannot retract streamed
-   content, plus the exposure bound when the host mediates
-   incrementally under the §12.1 exception.
+   content, plus the exposure bound and the `incremental_output`
+   declaration (its report then covers the `streaming/incremental`
+   part) when the host mediates incrementally under the §12.1
+   exception.
 
 Acceptance is by CODEOWNERS review (`conformance/` owner). The
 reviewer checks: the report matches the declared surface tuple; the

conformance/HARNESS.md

@@ -106,18 +106,69 @@ coercion class the core's raw-text scan (§10.2) exists to reject; see
 `true` (the host buffers caller-bound output until the `output`
 combined verdict permits), and a host that streams to its caller
 without buffering declares `buffered_output: false` in its surface and
-claim. A declaring host may also mediate `post_model_call`
-incrementally under the §12.1 exception; its claim then states the
-exposure bound (§12.1a). The CTK drives hosts with mocked I/O and
-cannot exercise streaming egress, so no vector carries this capability
-— the declaration exists to make the retraction limitation visible
-(§13.3). Vectors exercising the §12.1 accounting discipline against a
-mocked stream are future work.
+claim. The CTK drives hosts with mocked I/O and cannot exercise
+streaming egress, so no vector carries this capability — the
+declaration exists to make the retraction limitation visible (§13.3).
+
+`incremental_output` gates the `streaming/incremental` vector part
+(`AH-CTK-110`–`AH-CTK-113`), which exercises the §12.1 exception's
+accounting discipline against a mocked stream (see "Incremental
+mediation" below). Declare it only when your host declares
+`buffered_output: false` **and** mediates `post_model_call`
+incrementally with watermark-gated release (declared exposure bound:
+none — the posture ACS §18.1 calls `blocking`); the vectors pin that
+posture's deterministic release points, so a host with a looser bound
+does not declare the capability and skips the part. A buffering host
+never declares it.
 
 Non-finite floats (NaN/Infinity) and lone surrogates cannot be
 expressed in a JSON vector at all — those §4.4 marshalling guards are
 pinned by per-SDK unit tests, not vectors.
 
+## Incremental mediation
+
+Vectors in the `streaming/incremental` part carry a chunked mock
+stream: `respond.stream` is an ordered list of chunks whose
+concatenation equals `respond.content`. A harness declaring
+`incremental_output` MUST drive them as follows:
+
+- The mock model delivers the chunks in order. Each chunk boundary
+  closes one evaluated segment, and the host emits one
+  `post_model_call` per segment over the **assembled prefix** through
+  that chunk: `response.content` is the prefix, `response.finish_reason`
+  is `"incremental"` for a non-final segment and the scripted
+  `finish_reason` for the final one. Each emission is an ordinary
+  `post_model_call` (§12.1); the scripted interceptor answers each.
+- Release is watermark-gated: a permitted segment's text egresses on
+  its verdict; a `deny` terminates the stream, withholds everything
+  not yet released, and stops chunk consumption.
+- `respond.stream_truncated: true` means the stream dies abnormally
+  after the listed chunks: the final chunk is a partial segment no
+  emission covers, and the scripted `finish_reason` never arrives. The
+  host MUST fail closed per §12.1 exception item 3 — a
+  `post_model_call` over the full delivered assembly with
+  `response.finish_reason: "stream_incomplete"` and a deny
+  self-verdict `host_error:streaming_unsupported` — withholding and
+  not persisting the residue. The vectors assert the record, not
+  whether interceptors observe that emission (host-defined, as with
+  provider faults).
+- The `RunRecord` grows two fields for this part: `released_output` —
+  the caller-visible content the host actually egressed, in order
+  (distinct from `final_output`, which stays null on a blocked run) —
+  and `persisted` — a serialization of every durable incorporation the
+  host made for the session (conversation history, session stores).
+  `expect.released_output` is compared exactly;
+  `expect.persisted_must_not_contain` asserts substrings that must not
+  appear in `persisted`. A host that persists nothing reports an empty
+  value and satisfies the durability assertions vacuously, which is
+  the always-safe §6.1 posture.
+
+No in-tree reference harness declares `incremental_output` yet (every
+reference harness buffers), so runner-side assertion support for
+`released_output`/`persisted_must_not_contain` lands with the first
+declaring host; the part is capability-gated precisely so it stays
+inert for every buffered surface until then.
+
 ## Running
 
 ```bash
@@ -151,9 +202,14 @@ is pinned by per-SDK unit tests instead:
 
 - **NaN/Infinity marshalling guards (§4.4)** — not representable in a
   JSON vector file.
-- **§12.1 streaming assembly** — the scenario grammar has no partial-
-  stream form; the fail-closed `host_error:streaming_unsupported` path
-  is a host obligation the mocked model cannot exercise.
+- **§12.1 streaming assembly (buffered path)** — for a buffering host
+  the scenario grammar has no partial-stream form, so the
+  assemble-before-`post_model_call` rule and its fail-closed
+  `host_error:streaming_unsupported` path remain host obligations the
+  mocked model cannot exercise. The *incremental* path is different:
+  the `streaming/incremental` part drives it through `respond.stream`
+  for hosts declaring `incremental_output` (see "Incremental
+  mediation"), including the residue fail-closed shape (`AH-CTK-112`).
 - **§12.2 concurrent emissions** — vectors run single-threaded;
   sequence-uniqueness under concurrency is a per-SDK unit test.
 - **Multi-turn sessions (§3.1)** — the scenario grammar carries one

conformance/vectors.schema.json

@@ -45,9 +45,10 @@
           "multi_turn",
           "int64_json",
           "bigint_json",
-          "buffered_output"
+          "buffered_output",
+          "incremental_output"
         ],
-        "description": "int64_json marks vectors whose JSON carries integers beyond 2^53 (within i64/u64): a JavaScript harness cannot even load such a file without silent rounding (JSON.parse), so it omits the capability and skips. bigint_json marks vectors carrying integers beyond u64/i64, which additionally exceed serde_json's integer range — only harness languages whose JSON layer preserves raw numeric tokens (Python int, Go json.Number, .NET JsonNode) can claim it. buffered_output is declaration-only (§12.1a): it appears in a host's declared surface and claim, never as a vector requirement — the CTK's mocked I/O cannot exercise streaming egress."
+        "description": "int64_json marks vectors whose JSON carries integers beyond 2^53 (within i64/u64): a JavaScript harness cannot even load such a file without silent rounding (JSON.parse), so it omits the capability and skips. bigint_json marks vectors carrying integers beyond u64/i64, which additionally exceed serde_json's integer range — only harness languages whose JSON layer preserves raw numeric tokens (Python int, Go json.Number, .NET JsonNode) can claim it. buffered_output is declaration-only (§12.1a): it appears in a host's declared surface and claim, never as a vector requirement — the CTK's mocked I/O cannot exercise streaming egress. incremental_output marks the streaming/incremental vector part (§12.1 exception): the host declares buffered_output: false, mediates post_model_call incrementally with watermark-gated release (declared exposure bound: none), and its harness can drive a chunked mock stream (scenario respond.stream) and report released_output and persisted content. A host that buffers (buffered_output: true, the default) does not declare it and skips these vectors."
       },
       "default": []
     },
@@ -221,6 +222,19 @@
                   "finish_reason"
                 ],
                 "properties": {
+                  "stream": {
+                    "type": "array",
+                    "items": {
+                      "type": "string"
+                    },
+                    "minItems": 1,
+                    "description": "Chunked delivery of this response for an incremental_output host (§12.1 exception): the mock model delivers these chunks in order, each chunk boundary closes one evaluated segment, and the host emits one post_model_call per segment over the assembled prefix through that chunk (response.content is the prefix; response.finish_reason is \"incremental\" for a non-final segment and this respond's finish_reason for the final one). The chunk concatenation MUST equal content. Ignored by buffered hosts, which never run a vector carrying incremental_output."
+                  },
+                  "stream_truncated": {
+                    "type": "boolean",
+                    "default": false,
+                    "description": "The mock stream terminates abnormally after delivering the listed chunks: the final chunk is a partial segment no emission covers and the scripted finish_reason never arrives. The host has residue no emission evaluated and MUST fail closed per §12.1 exception item 3 — a post_model_call over the full delivered assembly with response.finish_reason \"stream_incomplete\" and a deny self-verdict host_error:streaming_unsupported — withholding and not persisting the residue. Only meaningful with stream."
+                  },
                   "content": true,
                   "tool_calls": {
                     "type": "array",
@@ -419,6 +433,16 @@
           }
         },
         "final_output": true,
+        "released_output": {
+          "description": "Caller-visible content the host actually egressed, in order, by the end of the run (streaming/incremental part only). Asserted against RunRecord.released_output, which the harness of an incremental_output host MUST report. Distinct from final_output: a stream denied mid-way returns no final output, yet the prefix released under covering verdicts had already reached the caller and is pinned here."
+        },
+        "persisted_must_not_contain": {
+          "type": "array",
+          "items": {
+            "type": "string"
+          },
+          "description": "Substrings that MUST NOT appear anywhere in the serialization of RunRecord.persisted — the harness-reported durable incorporations the host made for this session (conversation history, session stores; §6.1, §12.1 exception item 4). A host that persists nothing reports an empty value and passes vacuously, which is the always-safe §6.1 posture; the assertion pins that withheld or unevaluated stream content never became durable."
+        },
         "run_outcome": {
           "type": "string",
           "enum": [

conformance/vectors/AH-CTK-110-incremental-release-within-bound.json

@@ -0,0 +1,91 @@
+{
+  "id": "AH-CTK-110",
+  "title": "incremental mediation: a fully permitted stream is released segment by segment, each release under a covering verdict (§12.1 exception item 1; the covering-verdict bite on a denied stream is AH-CTK-111)",
+  "spec_clause": "12.1, 12.1a",
+  "part": "streaming/incremental",
+  "capabilities": [
+    "model_calls",
+    "incremental_output"
+  ],
+  "scenario": {
+    "input": {
+      "content": "stream a reply",
+      "role": "user"
+    },
+    "model_script": [
+      {
+        "respond": {
+          "content": "the quick brown fox",
+          "stream": [
+            "the quick ",
+            "brown fox"
+          ],
+          "tool_calls": [],
+          "finish_reason": "stop"
+        }
+      }
+    ]
+  },
+  "interceptor_script": [
+    {
+      "at": "post_model_call",
+      "return": {
+        "decision": "allow"
+      }
+    }
+  ],
+  "expect": {
+    "sequence_strict": true,
+    "interceptions": [
+      {
+        "interception_point": "agent_startup"
+      },
+      {
+        "interception_point": "input"
+      },
+      {
+        "interception_point": "pre_model_call"
+      },
+      {
+        "interception_point": "post_model_call",
+        "context": {
+          "response.content": "the quick ",
+          "response.finish_reason": "incremental"
+        }
+      },
+      {
+        "interception_point": "post_model_call",
+        "context": {
+          "response.content": "the quick brown fox",
+          "response.finish_reason": "stop"
+        }
+      },
+      {
+        "interception_point": "output",
+        "context": {
+          "output.content": "the quick brown fox"
+        }
+      },
+      {
+        "interception_point": "agent_shutdown"
+      }
+    ],
+    "records": [
+      {
+        "interception_point": "post_model_call",
+        "assert": {
+          "verdict.decision": "allow"
+        }
+      },
+      {
+        "interception_point": "post_model_call",
+        "assert": {
+          "verdict.decision": "allow"
+        }
+      }
+    ],
+    "released_output": "the quick brown fox",
+    "final_output": "the quick brown fox",
+    "run_outcome": "completed"
+  }
+}

conformance/vectors/AH-CTK-111-incremental-deny-withholds-remainder.json

@@ -0,0 +1,91 @@
+{
+  "id": "AH-CTK-111",
+  "title": "incremental mediation: a terminating deny withholds the unreleased remainder — the denied segment and every chunk behind it, harmless or not — while the prefix released under covering verdicts stays released (§12.1 exception items 1–2; with the watermark-gated bound a permitted segment is released on its verdict, so the withheld-but-permitted clause of item 2 is structurally empty here and is exercised on the durable side by AH-CTK-113)",
+  "spec_clause": "12.1, 12.1a",
+  "part": "streaming/incremental",
+  "capabilities": [
+    "model_calls",
+    "incremental_output"
+  ],
+  "scenario": {
+    "input": {
+      "content": "stream a reply",
+      "role": "user"
+    },
+    "model_script": [
+      {
+        "respond": {
+          "content": "public part, SECRET-PART then a harmless tail",
+          "stream": [
+            "public part, ",
+            "SECRET-PART",
+            " then a harmless tail"
+          ],
+          "tool_calls": [],
+          "finish_reason": "stop"
+        }
+      }
+    ]
+  },
+  "interceptor_script": [
+    {
+      "at": "post_model_call",
+      "match": {
+        "response.content": "public part, SECRET-PART"
+      },
+      "return": {
+        "decision": "deny",
+        "reason": "ctk:streamed-secret"
+      }
+    },
+    {
+      "at": "post_model_call",
+      "return": {
+        "decision": "allow"
+      }
+    }
+  ],
+  "expect": {
+    "sequence_strict": false,
+    "interceptions": [
+      {
+        "interception_point": "post_model_call",
+        "context": {
+          "response.content": "public part, ",
+          "response.finish_reason": "incremental"
+        }
+      },
+      {
+        "interception_point": "post_model_call",
+        "context": {
+          "response.content": "public part, SECRET-PART",
+          "response.finish_reason": "incremental"
+        }
+      },
+      {
+        "interception_point": "agent_shutdown"
+      }
+    ],
+    "interceptions_absent": [
+      "output"
+    ],
+    "records": [
+      {
+        "interception_point": "post_model_call",
+        "assert": {
+          "verdict.decision": "allow"
+        }
+      },
+      {
+        "interception_point": "post_model_call",
+        "assert": {
+          "verdict.decision": "deny",
+          "verdict.reason": "ctk:streamed-secret"
+        }
+      }
+    ],
+    "released_output": "public part, ",
+    "final_output": null,
+    "run_outcome": "blocked"
+  }
+}