Skip to content

feat: add policy YAML validation command to CLI#665

Open
kanish5 wants to merge 2 commits intomicrosoft:mainfrom
kanish5:feat/policy-yaml-validate
Open

feat: add policy YAML validation command to CLI#665
kanish5 wants to merge 2 commits intomicrosoft:mainfrom
kanish5:feat/policy-yaml-validate

Conversation

@kanish5
Copy link
Copy Markdown
Contributor

@kanish5 kanish5 commented Apr 1, 2026

Closes #529

Summary

Added agentos policy validate <file> CLI command that:

  1. Parses the YAML file (reports syntax errors with line numbers from PyYAML)
  2. Validates against the policy JSON schema via jsonschema (best-effort)
  3. Runs Pydantic structural validation via existing PolicyDocument
  4. Reports errors with field locations (e.g. rules[2] -> action)
  5. Exits with non-zero code on failure (CI-friendly)

New commands

  • agentos policy validate <file> — validate a single policy file
  • agentos policy test <policy> <scenarios> — run scenario tests
  • agentos policy diff <file1> <file2> — compare two policies

Changes

  • packages/agent-os/src/agent_os/cli/__init__.py:
    • Enhanced cmd_validate with JSON Schema + field location error reporting
    • Added _validate_yaml_with_line_numbers() helper
    • Added _load_json_schema() helper
    • Added cmd_policy() dispatcher
    • Registered agentos policy subcommand in the CLI

@github-actions
Copy link
Copy Markdown

github-actions bot commented Apr 1, 2026

Welcome to the Agent Governance Toolkit! Thanks for your first pull request.
Please ensure tests pass, code follows style (ruff check), and you have signed the CLA.
See our Contributing Guide.

@github-actions
Copy link
Copy Markdown

github-actions bot commented Apr 1, 2026

🤖 AI Agent: breaking-change-detector — Summary

🔍 API Compatibility Report

Summary

This pull request introduces new CLI commands and enhances existing functionality for policy validation and governance. It includes additive changes, such as new public APIs, helper functions, and CLI subcommands. No breaking changes were detected in the diff.

Findings

Severity Package Change Impact
🔵 agent-os Added cmd_policy() dispatcher New CLI command for policy management.
🔵 agent-os Added _validate_yaml_with_line_numbers() helper New internal helper for YAML validation.
🔵 agent-os Added _load_json_schema() helper New internal helper for loading JSON schemas.
🔵 agent-os Added agentos policy validate, test, and diff CLI commands New CLI commands for policy validation, testing, and comparison.
🔵 agent-os Added governed wrapper methods in crewai_adapter.py New methods for policy enforcement in CrewAI integrations.
🔵 agent-os Added docstrings for existing methods in google_adk_adapter.py Improves documentation without changing functionality.

Migration Guide

No migration steps are necessary as no breaking changes were identified.

Conclusion

No breaking changes detected. This pull request is safe to merge from an API compatibility perspective.

@github-actions github-actions bot added the size/XL Extra large PR (500+ lines) label Apr 1, 2026
@github-actions
Copy link
Copy Markdown

github-actions bot commented Apr 1, 2026

🤖 AI Agent: docs-sync-checker — Issues Found

📝 Documentation Sync Report

Issues Found

  1. cmd_policy() in agent_os/cli/__init__.py — missing docstring.
  2. _load_json_schema() in agent_os/cli/__init__.py — missing type hints.
  3. _validate_yaml_with_line_numbers() in agent_os/cli/__init__.py — missing type hints.
  4. ⚠️ packages/agent-os/README.md — does not mention the new agentos policy commands (validate, test, diff).
  5. ⚠️ CHANGELOG.md — no entry for the addition of the agentos policy commands and their functionality.

Suggestions

  • 💡 Add a docstring for cmd_policy() explaining its purpose, parameters, return values, and exceptions.
  • 💡 Add type hints for _load_json_schema() and _validate_yaml_with_line_numbers().
  • 💡 Update packages/agent-os/README.md to include a new section or update the CLI commands section to document the agentos policy commands (validate, test, diff) with examples of their usage.
  • 💡 Add an entry to CHANGELOG.md summarizing the new agentos policy commands and their features.

Additional Notes

  • The new docstrings added to methods in crewai_adapter.py and google_adk_adapter.py are well-written and provide clear explanations of the purpose, parameters, return values, and exceptions. Great job!
  • Ensure that the example code in the examples/ directory (if applicable) is updated to demonstrate the usage of the new agentos policy commands.

Please address the above issues and suggestions to ensure the documentation is fully in sync with the code changes.

Copy link
Copy Markdown

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 AI Agent: code-reviewer

Review Summary

This pull request introduces a new CLI command, agentos policy validate, which validates YAML policy files against a JSON schema and performs Pydantic validation. It also includes new interactive Jupyter notebooks for demonstrating policy enforcement, MCP security, and multi-agent governance. While the PR introduces valuable features, there are some critical security concerns, potential breaking changes, and areas for improvement.


🔴 CRITICAL

  1. Insufficient Validation of YAML Input

    • Issue: The _validate_yaml_with_line_numbers() function does not sanitize or validate the YAML input for potential malicious payloads. YAML files can include arbitrary Python objects, which could lead to code execution vulnerabilities.
    • Impact: This could allow an attacker to execute arbitrary code by injecting malicious payloads into the YAML file.
    • Recommendation: Use yaml.safe_load() instead of yaml.load() to prevent the execution of arbitrary code. If safe_load() is already being used, ensure that it is explicitly mentioned in the code or documentation.
  2. Potential Bypass of JSON Schema Validation

    • Issue: The JSON schema validation is described as "best-effort," which implies that it might not catch all invalid inputs. This could lead to security vulnerabilities if the policy engine relies on the validity of the schema for enforcement.
    • Impact: A malicious or malformed policy file could bypass validation and introduce vulnerabilities into the system.
    • Recommendation: Ensure that the JSON schema is comprehensive and rigorously tested. Consider adding unit tests to validate edge cases and ensure the schema is robust.
  3. Regex Injection in MCP Security Scanner

    • Issue: The scan_tool_definition function uses user-provided input (description) directly in regex operations. If the regex patterns are not properly sanitized, this could lead to regex injection vulnerabilities.
    • Impact: An attacker could craft a malicious tool description to exploit the regex engine, potentially causing denial of service (ReDoS) or other unexpected behavior.
    • Recommendation: Use a library like re.escape() to sanitize user-provided input before using it in regex operations.
  4. Lack of Thread Safety in Circuit Breaker Implementation

    • Issue: The simulate_call and check_circuit_breaker functions modify shared AgentMetrics objects without any synchronization mechanisms. This could lead to race conditions in a multi-threaded environment.
    • Impact: Inconsistent or incorrect metrics could lead to incorrect circuit breaker behavior, potentially allowing agents to operate in unsafe conditions.
    • Recommendation: Use thread-safe data structures or synchronization primitives (e.g., threading.Lock) to ensure thread safety when modifying shared state.

🟡 WARNING

  1. Backward Compatibility
    • Issue: The addition of the agentos policy subcommand introduces new functionality to the CLI. While this is not a breaking change, it is important to ensure that existing commands and workflows are not affected.
    • Recommendation: Verify that the new subcommand does not interfere with existing CLI commands. Add tests to ensure backward compatibility.

💡 SUGGESTIONS

  1. Error Reporting

    • Observation: The error reporting for the agentos policy validate command is functional but could be improved for user experience.
    • Suggestion: Consider adding color-coded output (e.g., using the colorama library) to make errors and warnings more visually distinct. Additionally, provide suggestions for fixing common errors when possible.
  2. Documentation

    • Observation: The new CLI commands and Jupyter notebooks are valuable additions, but the documentation does not appear to have been updated to reflect these changes.
    • Suggestion: Update the README and any relevant documentation to include details about the new CLI commands and how to use the Jupyter notebooks.
  3. Testing

    • Observation: While the PR mentions that the agentos policy validate command is "CI-friendly," there is no evidence of automated tests for this functionality.
    • Suggestion: Add unit tests and integration tests for the new CLI commands, especially for edge cases in YAML parsing, JSON schema validation, and Pydantic validation.
  4. Notebook Security

    • Observation: The Jupyter notebooks include code that simulates potentially dangerous operations (e.g., exec('/bin/sh') in the MCP Security Proxy notebook).
    • Suggestion: Add clear warnings in the notebook headers to indicate that the code is for educational purposes only and should not be used in production environments without proper safeguards.
  5. Code Quality

    • Observation: The _validate_yaml_with_line_numbers() and _load_json_schema() helper functions are not documented.
    • Suggestion: Add docstrings to these functions to improve code readability and maintainability.
  6. Error Handling

    • Observation: The agentos policy validate command exits with a non-zero code on failure, but the specific exit codes are not documented.
    • Suggestion: Document the possible exit codes and their meanings in the CLI help text or documentation.

Conclusion

This PR introduces useful features and enhancements to the agentos CLI and the Jupyter notebooks. However, there are critical security issues that must be addressed before merging. Additionally, improvements to documentation, testing, and error reporting would enhance the overall quality of the contribution.

@github-actions
Copy link
Copy Markdown

github-actions bot commented Apr 1, 2026

🤖 AI Agent: test-generator — `packages/agent-os/src/agent_os/cli/__init__.py`

🧪 Test Coverage Analysis

packages/agent-os/src/agent_os/cli/__init__.py

  • ✅ Existing coverage: The CLI command for validating policy YAML files is likely covered by existing tests that check command-line interface functionality, including argument parsing and error handling.
  • ❌ Missing coverage: There may be insufficient tests for edge cases in policy validation, such as:
    • Handling of invalid YAML syntax with line numbers.
    • Validation against the JSON schema, especially for missing required fields and invalid field types.
    • Strict mode checks for unknown top-level fields.
    • Behavior when no files are found or when files are empty.
  • 💡 Suggested test cases:
    1. test_validate_yaml_syntax_error — Test the CLI command with a malformed YAML file to ensure it reports the correct syntax error with line numbers.
    2. test_validate_missing_required_fields — Test the CLI command with a YAML file missing required fields to verify it correctly identifies and reports these errors.
    3. test_validate_invalid_action_type — Test the CLI command with a YAML file containing invalid action types in rules to ensure it reports the appropriate validation errors.
    4. test_validate_strict_mode_unknown_fields — Test the CLI command with a YAML file containing unknown top-level fields in strict mode to check that these are reported as warnings.

packages/agent-os/src/agent_os/integrations/crewai_adapter.py

  • ✅ Existing coverage: The integration with CrewAI tools likely has some existing tests that validate the basic functionality of the governed wrappers.
  • ❌ Missing coverage: There may be gaps in testing for:
    • Policy violation scenarios, particularly how the governed wrappers handle violations.
    • Edge cases in input validation for the governed methods, such as unexpected argument types or missing required arguments.
    • Concurrency issues when multiple agents attempt to execute tasks simultaneously.
  • 💡 Suggested test cases:
    1. test_governed_run_policy_violation — Simulate a policy violation during a tool's run method and verify that the appropriate exception is raised and logged.
    2. test_governed_execute_invalid_task — Test the governed execution method with an invalid task object to ensure it handles the error gracefully.
    3. test_governed_step_concurrent_execution — Create a scenario where multiple steps are executed concurrently to check for race conditions or deadlocks.
    4. test_governed_save_pii_detection — Test the governed save method with content that includes PII to ensure it correctly raises a policy violation.

packages/agent-os/src/agent_os/integrations/google_adk_adapter.py

  • ✅ Existing coverage: The integration with the ADK likely has tests that cover basic functionality and error handling.
  • ❌ Missing coverage: Potential gaps include:
    • Handling of blocked tools and patterns, especially edge cases where tools are conditionally allowed or blocked.
    • Timeout handling and how the system behaves when timeouts are exceeded.
    • Logging behavior for policy violations and audit events.
  • 💡 Suggested test cases:
    1. test_check_tool_blocked — Test the tool check method with a blocked tool to ensure it returns the correct response indicating the tool is blocked.
    2. test_check_content_blocked_pattern — Test the content check method with text that matches a blocked pattern to verify it raises the appropriate violation.
    3. test_timeout_handling — Simulate a scenario where the kernel exceeds its timeout and ensure the system responds correctly, possibly by raising an exception or logging an error.
    4. test_audit_logging_on_violation — Trigger a policy violation and verify that the violation is logged correctly in the audit log.

packages/agent-os/src/agent_os/integrations/guardrails_adapter.py

  • ✅ Existing coverage: There may be tests that cover the basic functionality of the guardrails integration.
  • ❌ Missing coverage: Potential gaps include:
    • Edge cases in policy enforcement, such as conflicting policies or bypass attempts.
    • Input validation for the guardrails, especially malformed inputs or oversized payloads.
  • 💡 Suggested test cases:
    1. test_conflicting_policies_enforcement — Create a scenario with conflicting policies and verify that the system correctly handles the conflict.
    2. test_input_validation_malformed — Test the guardrails with malformed inputs to ensure it raises appropriate validation errors.
    3. test_input_validation_oversized_payload — Test the guardrails with oversized payloads to check for proper handling and error reporting.

packages/agent-os/src/agent_os/integrations/pydantic_ai_adapter.py

  • ✅ Existing coverage: Basic functionality of the Pydantic integration is likely covered.
  • ❌ Missing coverage: Gaps may include:
    • Edge cases in model validation, particularly with complex nested structures.
    • Handling of expired or revoked trust scenarios.
  • 💡 Suggested test cases:
    1. test_model_validation_nested_structure — Test the Pydantic model with complex nested data to ensure it validates correctly.
    2. test_expired_certificate_handling — Simulate a scenario where a certificate is expired and verify that the system responds appropriately.
    3. test_revoked_trust_handling — Test the behavior when trust is revoked and ensure that the system correctly identifies and handles this case.

packages/agent-os/src/agent_os/integrations/smolagents_adapter.py

  • ✅ Existing coverage: Basic functionality of the SmolAgents integration is likely covered.
  • ❌ Missing coverage: Potential gaps include:
    • Concurrency issues when multiple agents interact with shared resources.
    • Edge cases in policy enforcement and input validation.
  • 💡 Suggested test cases:
    1. test_concurrent_agent_interaction — Simulate multiple agents interacting with shared resources to check for race conditions or deadlocks.
    2. test_policy_enforcement_edge_cases — Create scenarios that test edge cases in policy enforcement to ensure the system behaves as expected.
    3. test_input_validation_injection_attempts — Test the input validation for potential injection attempts to ensure the system is secure against such threats.

@github-actions
Copy link
Copy Markdown

github-actions bot commented Apr 1, 2026

🤖 AI Agent: security-scanner — Security Review of PR: `feat: add policy YAML validation command to CLI`

Security Review of PR: feat: add policy YAML validation command to CLI

This pull request introduces a new CLI command, agentos policy validate <file>, which validates YAML policy files against a JSON schema, performs structural validation using Pydantic, and provides detailed error reporting. While this feature is highly useful, it also introduces potential security risks that need to be addressed. Below is a detailed review of the changes with a focus on the specified security concerns.


Findings

1. Prompt Injection Defense Bypass

  • Risk: The _validate_yaml_with_line_numbers function does not sanitize or validate the content of the YAML file for potential prompt injection attacks. If the policy file contains malicious content that is later used in prompts or agent interactions, it could lead to prompt injection vulnerabilities.
  • Attack Vector: An attacker could craft a malicious policy file containing specially crafted strings that exploit downstream AI agent prompts.
  • Severity: 🔴 CRITICAL
  • Recommendation:
    • Add a sanitization step to _validate_yaml_with_line_numbers to detect and reject potentially dangerous strings.
    • Use a strict allowlist approach for acceptable input formats and values.
    • Consider integrating a prompt sanitization library to detect and mitigate injection risks.

2. Policy Engine Circumvention

  • Risk: The validation logic in _validate_yaml_with_line_numbers does not enforce strict adherence to the JSON schema if the jsonschema library is not installed. This fallback behavior could allow invalid policies to bypass validation.
  • Attack Vector: An attacker could exploit environments where jsonschema is not installed to deploy policies that bypass critical validation checks.
  • Severity: 🟠 HIGH
  • Recommendation:
    • Make jsonschema a mandatory dependency for the project to ensure schema validation is always performed.
    • Alternatively, raise an error if jsonschema is not installed, rather than silently skipping schema validation.

3. Trust Chain Weaknesses

  • Risk: The _load_json_schema function reads the JSON schema from a file without verifying its integrity. If the file is tampered with, it could lead to incorrect validation.
  • Attack Vector: An attacker with access to the file system could modify the policy_schema.json file to weaken or bypass policy validation.
  • Severity: 🟠 HIGH
  • Recommendation:
    • Use a cryptographic hash to verify the integrity of the policy_schema.json file before using it.
    • Consider embedding the schema directly into the codebase or fetching it securely from a trusted source.

4. Credential Exposure

  • Risk: The CLI outputs detailed error messages, including the content of the YAML files, to the console. If these logs are captured in a shared environment or CI/CD pipeline, they could expose sensitive information.
  • Attack Vector: Sensitive information (e.g., API keys, secrets) embedded in policy files could be leaked through logs.
  • Severity: 🟠 HIGH
  • Recommendation:
    • Mask sensitive fields (e.g., api_key, password) in error messages and logs.
    • Provide a configuration option to suppress detailed error output in CI/CD environments.

5. Sandbox Escape

  • Risk: No evidence of sandboxing or isolation for the YAML validation process. If the YAML file contains malicious payloads (e.g., Python objects), it could exploit deserialization vulnerabilities.
  • Attack Vector: An attacker could craft a YAML file that exploits unsafe deserialization in yaml.safe_load.
  • Severity: 🔴 CRITICAL
  • Recommendation:
    • Use yaml.safe_load exclusively (already implemented) and ensure no unsafe loaders (e.g., yaml.load) are used elsewhere in the codebase.
    • Consider using a stricter YAML parser that does not support arbitrary Python objects.

6. Deserialization Attacks

  • Risk: The use of yaml.safe_load mitigates most deserialization risks, but there is no explicit check for unexpected data types or structures.
  • Attack Vector: A malicious actor could craft a YAML file with unexpected structures that cause the application to behave unpredictably.
  • Severity: 🟡 MEDIUM
  • Recommendation:
    • Add explicit checks for expected data types and structures after parsing the YAML file.
    • Use a schema validation library (e.g., cerberus or pydantic) to enforce strict type and structure constraints.

7. Race Conditions

  • Risk: The _load_json_schema function reads the schema file from disk at runtime. If the file is modified during execution, it could lead to inconsistent validation results.
  • Attack Vector: An attacker could exploit a time-of-check-to-time-of-use (TOCTOU) vulnerability by modifying the schema file between its loading and use.
  • Severity: 🟡 MEDIUM
  • Recommendation:
    • Load the schema file once at application startup and cache it in memory.
    • Use file locking mechanisms to prevent concurrent modifications.

8. Supply Chain Risks

  • Risk: The PR introduces a dependency on jsonschema, which could be a vector for supply chain attacks if a malicious version is published.
  • Attack Vector: An attacker could publish a malicious version of jsonschema to a public package repository, which could then be inadvertently included in the project.
  • Severity: 🟡 MEDIUM
  • Recommendation:
    • Pin dependencies to specific versions in requirements.txt or pyproject.toml.
    • Use a dependency scanning tool (e.g., Dependabot, Snyk) to monitor for vulnerabilities in dependencies.

Summary of Findings

Category Severity Description
Prompt Injection Defense Bypass 🔴 CRITICAL No sanitization of YAML content for prompt injection risks.
Policy Engine Circumvention 🟠 HIGH Validation bypass possible if jsonschema is not installed.
Trust Chain Weaknesses 🟠 HIGH No integrity verification for policy_schema.json.
Credential Exposure 🟠 HIGH Potential exposure of sensitive data in error logs.
Sandbox Escape 🔴 CRITICAL No isolation for YAML validation process.
Deserialization Attacks 🟡 MEDIUM Lack of explicit checks for unexpected data types or structures.
Race Conditions 🟡 MEDIUM TOCTOU vulnerability in _load_json_schema.
Supply Chain Risks 🟡 MEDIUM Dependency on jsonschema introduces potential supply chain risks.

Actionable Recommendations

  1. Prompt Injection Defense:

    • Sanitize and validate YAML content for potentially malicious strings.
    • Use a prompt sanitization library to detect injection risks.
  2. Policy Engine Hardening:

    • Make jsonschema a mandatory dependency or raise an error if unavailable.
    • Enforce strict validation against the JSON schema.
  3. Trust Chain Improvements:

    • Verify the integrity of policy_schema.json using a cryptographic hash.
    • Consider embedding the schema directly in the codebase.
  4. Credential Protection:

    • Mask sensitive fields in error messages and logs.
    • Provide an option to suppress detailed error output in CI/CD environments.
  5. Sandboxing:

    • Ensure the YAML validation process is isolated from the rest of the system.
    • Use secure YAML parsers and avoid deserialization of arbitrary objects.
  6. Race Condition Mitigation:

    • Cache the schema file in memory at application startup.
    • Use file locking to prevent concurrent modifications.
  7. Supply Chain Security:

    • Pin dependencies to specific versions.
    • Use dependency scanning tools to monitor for vulnerabilities.

Overall Assessment

This PR introduces a valuable feature for validating policy files but also introduces critical security risks that need to be addressed before merging. The most pressing issues are the potential for prompt injection, policy circumvention, and sandbox escape. These must be resolved to ensure the integrity and security of the agentos toolkit.

@kanish5
Copy link
Copy Markdown
Contributor Author

kanish5 commented Apr 1, 2026

@microsoft-github-policy-service agree

@kanish5 kanish5 force-pushed the feat/policy-yaml-validate branch from 059e587 to 415c4af Compare April 1, 2026 00:34
Copy link
Copy Markdown

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 AI Agent: code-reviewer

Feedback on the Pull Request

🔴 CRITICAL Issues

  1. Improper Handling of jsonschema Import Failure:

    • In _validate_yaml_with_line_numbers, the jsonschema library is imported inside the function, and if the import fails, the code silently skips JSON Schema validation. This could lead to undetected policy violations if the library is not installed.
    • Recommendation: Make jsonschema a required dependency in the setup.py or pyproject.toml file. If this is not feasible, explicitly log an error and exit with a non-zero status if the library is not installed.
  2. Insufficient Validation of Policy Schema:

    • The _validate_yaml_with_line_numbers function performs some basic checks (e.g., required fields, rule structure), but these checks are not comprehensive. For example:
      • The rules field is only checked for being a list, but the individual rule objects are not fully validated against the schema.
      • The action field is validated against a hardcoded list of valid actions, but this list may not stay in sync with the JSON Schema.
    • Recommendation: Use jsonschema exclusively for schema validation and avoid duplicating validation logic in _validate_yaml_with_line_numbers. This ensures consistency and reduces the risk of discrepancies between the schema and the code.
  3. Potential for Sandbox Escape in YAML Parsing:

    • The yaml.safe_load function is used for parsing YAML files. While this is safer than yaml.load, it is still vulnerable to certain attacks, such as those exploiting YAML tags.
    • Recommendation: Use a library like ruamel.yaml or a stricter YAML parser to avoid potential sandbox escape vulnerabilities.
  4. Error Reporting for YAML Parsing:

    • The error messages for YAML parsing issues are not sanitized. If an attacker provides a malicious YAML file, the error message could potentially leak sensitive information or be used for injection attacks.
    • Recommendation: Sanitize error messages before displaying them to the user.

🟡 WARNING Issues

  1. Backward Compatibility Risk:

    • The addition of the agentos policy subcommand introduces a new CLI structure. If users were relying on the existing validate command, this change could break their workflows.
    • Recommendation: Provide a deprecation warning for the old validate command and ensure that it continues to work for a transitional period.
  2. Potential Breaking Changes in Policy Validation:

    • The new validation logic introduces stricter checks (e.g., required fields, rule structure). Policies that previously passed validation may now fail.
    • Recommendation: Clearly document these changes in the release notes and provide a migration guide for users to update their policies.

💡 Suggestions for Improvement

  1. Improve Error Messages:

    • The error messages in _validate_yaml_with_line_numbers are functional but could be more user-friendly. For example, instead of rules[2] -> action, consider using a more descriptive format like Rule 2: Invalid action field.
    • Recommendation: Use a consistent and user-friendly format for error messages.
  2. Add Unit Tests for CLI Commands:

    • The new agentos policy subcommands (validate, test, diff) should be thoroughly tested to ensure they work as expected.
    • Recommendation: Add unit tests for each subcommand, covering both success and failure scenarios.
  3. Thread Safety in Policy Validation:

    • The _validate_yaml_with_line_numbers function and related logic do not appear to have any thread safety issues, but this should be explicitly verified if the CLI is expected to handle concurrent requests (e.g., in a CI/CD pipeline).
    • Recommendation: Add tests to verify thread safety in concurrent execution scenarios.
  4. Support for JSON Input:

    • The agentos policy validate command currently supports YAML files but does not explicitly mention support for JSON files.
    • Recommendation: Add explicit support for JSON files and update the documentation accordingly.
  5. Improve Documentation:

    • The docstrings for the new functions and methods are detailed, but the overall documentation for the agentos policy subcommands could be improved.
    • Recommendation: Add examples and usage instructions for the new subcommands in the CLI documentation.
  6. Use of pathlib for File Handling:

    • The code uses Path.read_text() and Path.write_text() for file I/O, which is good. However, there are some instances where open() is still used (e.g., in cmd_validate).
    • Recommendation: Use pathlib consistently for file handling.
  7. Consider Adding a Dry-Run Mode:

    • For the agentos policy test and agentos policy diff subcommands, a dry-run mode could be useful for users who want to preview the results without making changes.
    • Recommendation: Add a --dry-run flag to these subcommands.
  8. Logging Improvements:

    • The CLI commands use print() for output, which is fine for user-facing messages but not ideal for logging errors or debugging information.
    • Recommendation: Use the logging module for error and debug messages.

Summary of Changes Needed

Type Description
🔴 CRITICAL Ensure jsonschema is a required dependency or handle its absence explicitly.
🔴 CRITICAL Use jsonschema exclusively for schema validation to avoid duplication.
🔴 CRITICAL Use a stricter YAML parser to prevent sandbox escape vulnerabilities.
🔴 CRITICAL Sanitize error messages to prevent information leakage or injection attacks.
🟡 WARNING Maintain backward compatibility for the validate command.
🟡 WARNING Document stricter validation rules and provide a migration guide.
💡 SUGGESTION Improve error messages, documentation, and add unit tests for new commands.
💡 SUGGESTION Add support for JSON input and a dry-run mode for certain subcommands.
💡 SUGGESTION Use pathlib consistently and improve logging practices.

By addressing these issues and suggestions, the new feature will be more robust, secure, and user-friendly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/XL Extra large PR (500+ lines)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: add policy YAML validation command to CLI

1 participant