[AAASM-4800] ✅ (test): Fail-open conformance matrix across adapters × scenarios × modes#276
Merged
Merged
Conversation
Fault-injection harness + DRIVERS registry that drives each adapter's real governance wrapper against a fake interceptor per scenario, reporting whether the underlying tool ran. Refs AAASM-4800.
…s × modes Parametrize every governing adapter × 8 fault scenarios × 4 enforcement modes, asserting no adapter fails open under enforce/None and pinning the observe/ disabled dry-run postures. A completeness test fails if a new/regressed adapter package ships without a driver or a justified exemption (langgraph, _shared). Named guards pin the openai_agents (AAASM-4782/4734) and langchain (AAASM-4790) regressions. Closes AAASM-4800.
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
|
Contributor
Author
🤖 Claude Code — PR reviewScope (AAASM-4800): systemic fail-open conformance — parametrized matrix (11 adapter drivers × 8 scenarios × 4 modes) + adapter-driver registry with a completeness guard.
Verdict: ready to merge pending Pioneer approval. This is the anti-whack-a-mole guard that ends the per-round fail-open findings. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds a parametrized fail-open CONFORMANCE test — every governing adapter × 8 fault scenarios × 4 enforcement modes — backed by an adapter-driver registry, so a new or regressed adapter cannot ship ungoverned. This is the systemic fix that ends the per-round fail-open whack-a-mole (openai_agents AAASM-4734 → AAASM-4782, langchain AAASM-4790): instead of one bespoke regression test each time a single adapter is caught failing open, all 11 governing adapters are driven through the same faults under the same postures.
How it works. Each adapter has a driver that builds a fake framework tool whose real body flips a
ranflag, invokes the adapter's real governance wrapper (_apply_*/ callback helper) against a fake interceptor whose behaviour is set per scenario, and reports whether the underlying tool actually ran.Invariant asserted. Under an enforce posture (explicit
enforceand theNone/omitted default — resolved via the production_local_posture_is_enforce, pinning that omitted → enforce), every fault (S1–S5) and every deny (S6–S7) blocks the tool; only the explicit-allow control (S8) runs. Under the observe / disabled dry-run postures the faults fail open by design (tool runs), while an authoritative deny still blocks — so the modes are pinned in both directions.Scenarios: S1 gateway-unreachable (no verdict), S2 governance/transport error, S3 malformed decision, S4 missing check method, S5 pending-approval fault, S6 deny + deny-audit raises, S7 explicit deny, S8 explicit allow.
Anti-whack-a-mole guard. A completeness test enumerates the adapter packages on disk and fails if any lacks a registered driver or a justified exemption (
langgraph— lineage-only, no pre-exec tool gate;_shared— shared helper, not a framework adapter). A brand-new adapter package therefore fails CI until it gets a driver.Type of Change
Breaking Changes
Related Issues
Testing
test/unit/adapters/test_failopen_conformance.py, harness infailopen_conformance.py)Local run of the new file only: 361 passed (352 matrix cells + completeness/disjoint guards + 6 named regression cells).
ruff check,ruff format, andmypyall clean on the new files. Wired into CI by placement —ci.yamlrunstest/**/*.py.Regression guards confirmed to PASS on this base (both fixes are merged to master) and confirmed non-vacuous: reintroducing the AAASM-4782 audit-guard bug makes the observe cell report
ran=True, which fails the guard.Checklist