feat(conformance): declared tool-seam posture and shape-agnostic AH-CTK-100 - #72
Merged
Merged
Conversation
…TK-100 Two CTK fixes so a spec-permitted host surface can make an honest $13.1 claim (found by the MAF cross-validation, conformance/claims/maf): - #68: hosts declare tool_seam_host_error: continue|terminate ($6.2's "unless the host's own semantics terminate the turn" clause). The 13 tool-seam host_error:* vectors gain expect.run_outcome_by_posture and the runner resolves it against the declaration, so every (vector, declared surface) pair stays single-valued. run_outcome keeps the default-posture value and the engine enforces their equality, so pre-posture runners (e.g. ACS's pinned alpha.3 engine) still resolve the corpus for continue hosts. - #69: AH-CTK-100 now asserts the $6.1/$6.2 substance — the denied tool result never surfaces at the next pre_model_call (context_must_not_contain), the deny reason surfaces in SOME form (context_must_contain), the tool ran once — instead of pinning the reference host's transcript cosmetics (message index and "blocked: <reason>" string). The posture declaration defaults to continue in all five SDK Harness contracts (attribute / optional interface / default member / trait default), matching every in-tree reference host; reference self-test expectations are unchanged by construction. Verified: Rust workspace, Python (181 passed), .NET (122 passed), TypeScript (131 passed) suites green locally; ajv validates all 51 vectors; ACS's conformance suite over the swapped corpus stays at 46 passed, 0 failed, 5 capability-gated skips. Closes #68 Closes #69 Signed-off-by: MohammadHaroonAbuomar <40180927+MohammadHaroonAbuomar@users.noreply.github.com>
MohammadHaroonAbuomar
requested a review
from Caitie McCaffrey (CaitieM20)
as a code owner
August 7, 2026 08:14
MohammadHaroonAbuomar
added
the
spec:additive
MohammadHaroonAbuomar
added a commit
that referenced
this pull request
Aug 7, 2026
…cable) Re-run of the Microsoft Agent Framework CTK suite against the corpus at 4f7af78 (post #72) with the harness declaring its tool_seam_host_error: terminate posture: 47 passed, 0 failed, 4 skipped of 51 The 4 skips are the capability-gated streaming/incremental part a buffered_output: true host never declares. The 13 former F1 vectors now pass under the declared posture; AH-CTK-100 (F2) passes with its shape-agnostic assertions; F3 (tools_registered projection, microsoft/agent-framework#7560) remains an observation costing no vector. REPORT.md rewritten as a claim (same adapter version, core 1.13.0 @ 4b1afd9052; agent-hooks-sdk 0.1.0a5 built from source at 4f7af78), harness gains the posture declaration, CLAIMS.md row upgraded from partial to conformant. Signed-off-by: MohammadHaroonAbuomar <40180927+MohammadHaroonAbuomar@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the two CTK issues blocking an honest MAF conformance claim (found by the cross-validation in
conformance/claims/maf/, PR #66).#68 — declared
tool_seam_host_errorposture§6.2 permits two behaviors after a
host_error:*deny at the tool seam: continue the loop (surface a tool error to the model) or terminate the turn ("unless the host's own semantics terminate the turn"). The CTK's single-valuedrun_outcomecould only express the first.Design — declared posture, not outcome sets: the harness declares
tool_seam_host_error: continue | terminate(defaultcontinue) alongside its capabilities, the runner forwards the declaration in the run-record wire (postures.tool_seam_host_error), and the engine resolves the newexpect.run_outcome_by_postureto the single outcome that declared surface must produce. A pass never means "one of several outcomes happened", so claims stay honest and vectors stay single-valued per declared surface.Backward compatibility is deliberate:
run_outcomekeeps the default-posture value and the engine enforces it equalsrun_outcome_by_posture.outcomes.continue, so pre-posture runners still resolve the corpus correctly for continue hosts — verified against agent-control-spec's pinnedagent-hooks-sdk 0.1.0-alpha.3engine (below).pre_tool_callhost_error:*denies).continuedefault: Python attribute, TypeScript optional readonly, Go optionalToolSeamHostErrorDeclarerinterface, .NET default interface member, Rust default trait method. Reference self-tests are unchanged by construction (every reference host continues).CLAIMS.mdrequire non-default postures to be stated in the claim.#69 — AH-CTK-100 asserts §6.1/§6.2 substance, not transcript cosmetics
The vector pinned the reference host's conventions (tool message at
messages[1], content exactly"blocked: <reason>"). It now asserts only the normative properties, via two new shape-agnostic interception assertions (context_must_contain/context_must_not_contain, substring scans over the serialized recorded context):SECRET-RESULT) never surfaces at the nextpre_model_callin any form (§6.1 non-incorporation),tool_invocations), plus the existing deny-record assertion.Verification
--all-features, incl. new engine unit tests)pytest sdk/python/tests)dotnet test)node --test)blockedCloses #68
Closes #69