test(T14-T15): governance extraction and audit engine tests#18
Closed
ogkranthi wants to merge 5 commits into
Closed
test(T14-T15): governance extraction and audit engine tests#18ogkranthi wants to merge 5 commits into
ogkranthi wants to merge 5 commits into
Conversation
… experiments) + clean up duplicate files - Add Governance, Guardrail, ToolPermission, PlatformAnnotation to IR model - Add governance extraction to OpenClaw parser (SOUL.md + tool permissions + L3 annotations) - Add elevation engine (elevate_governance) for L2/L3 → L1 promotion - Add governance_audit module with GPR/CFS scoring, CSV/JSON export, Rich tables - Add `agentshift audit` and `agentshift audit-batch` CLI commands - Integrate elevation into claude_code + copilot emitters - Add experiments/ directory with 12 domain agents for research paper - Remove duplicate sections 2.py and persona-sections-schema 2.md - Mark T13 as merged in BACKLOG.md
T14 — test_governance_extraction.py (68 tests): - GuardrailRule category/severity classification (all 7 categories, 4 severities) - ToolPermission field population: access, deny/allow patterns, rate_limit, max_value - PlatformAnnotation L3 parsing: all 4 kinds, all 4 platform targets, config dict - Edge cases: empty governance, missing required fields, extra fields rejected, defaults verified, combined layers T15 — test_governance_audit.py (73 tests): - GPR-L1 formula: always 1.0 (L1 always preserved), zero-guardrail default - GPR-L2 formula: native-only rate, elevated != preserved, platform matrix verified - GPR-L3 formula: zero default, bedrock full support, vertex partial, claude-code none - GPR-Overall: weighted by artifact count, cross-platform comparison - CFS: identity/tools/memory/schema checks, formula ratio - Elevation tracking: disabled tools, deny patterns, allow patterns, rate limits, L3 annotations; artifact keys, non-empty instructions, claude-code deny passthrough - CSV export: header exact match, parent dir creation, 4-decimal formatting, empty list - JSON export: structure, l1/l2/l3 subkeys, elevated_artifacts keys, empty list - audit_batch: agents×targets, empty cases - Edge cases: zero tools, all-denied copilot, perfect bedrock scores, agent_id defaults
Deploying agentshift with
|
| Latest commit: |
d47abfc
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://00a4fbc5.agentshift.pages.dev |
| Branch Preview URL: | https://agent-tester-t14-t15.agentshift.pages.dev |
…om bedrock/vertex, v0.3.0 D22 — AWS Bedrock parser (src/agentshift/parsers/bedrock.py) - Reads bedrock-agent.json, cloudformation.yaml, instruction.txt, openapi.json, guardrail-config.json (any combination; precedence: bedrock-agent.json > CFN > txt) - Reconstructs persona.system_prompt with section extraction - Extracts tools from OpenAPI action-group schemas (with CFN fallback) - Extracts knowledge sources from AWS::Bedrock::KnowledgeBase CFN resources - Strips AgentShift truncation notice from instruction.txt - Heuristic L1 guardrail extraction from instruction + guardrail-config.json topic policies - Registered under 'bedrock' parser key D23 — Vertex AI parser (src/agentshift/parsers/vertex.py) - Reads agent.json (required) + optional tools.json and README.md - Reconstructs system_prompt from goal + instructions with separator - Recovers structured sections from 'SectionName:\ncontent' linearized patterns - Detects tool kind: function / openapi / data store (routed to ir.knowledge) - Reconstructs ToolAuth from Vertex authentication blocks (apiKey/oauth/serviceAccount) - Heuristic L1 guardrail scan of instruction strings - Registered under 'vertex' parser key Shared utilities (src/agentshift/parsers/utils.py) - slugify, title_case_to_slug, is_todo_placeholder - infer_guardrail_category, infer_guardrail_severity - extract_guardrails_from_text (shared by both parsers) D24 — CLI updates (src/agentshift/cli.py) - Added 'bedrock' and 'vertex' to _PARSERS registry - convert/diff/audit now support --from bedrock and --from vertex - Enhanced _parse_with_errors with bedrock/vertex-specific error hints D25 — Version bump to 0.3.0 - pyproject.toml version: 0.3.0 - __init__.py __version__: 0.3.0 - CHANGELOG.md: v0.3.0 section added - README.md: governance layer docs + cloud parser examples Tests: 782 passed (all existing tests continue to pass)
Owner
Author
|
Tests cherry-picked to main directly (commit 27b27ca). T14-T15 merged. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes T14 and T15 (Week 7 governance framework tests).
T14 — Governance extraction tests (
tests/test_governance_extraction.py, 68 tests)T15 — Audit engine tests (
tests/test_governance_audit.py, 73 tests)[]Results
All 141 new tests pass. All 641 pre-existing tests continue to pass.