diff --git a/.github/workflows/pr_checks.yaml b/.github/workflows/pr_checks.yaml index 58fc2a9..1cd0588 100644 --- a/.github/workflows/pr_checks.yaml +++ b/.github/workflows/pr_checks.yaml @@ -62,7 +62,7 @@ jobs: - name: Clone repository with additional plugins and tests run: | - git clone --depth 1 ${{ secrets.ADDITIONAL_PLUGINS_REPO }} permian-redhat > /dev/null 2>&1 + git -c http.sslVerify=false clone --depth 1 ${{ secrets.ADDITIONAL_PLUGINS_REPO }} permian-redhat > /dev/null 2>&1 - name: Build main container run: | diff --git a/doc/development/howto-plugins.rst b/doc/development/howto-plugins.rst index 2bd690c..4cd95f9 100644 --- a/doc/development/howto-plugins.rst +++ b/doc/development/howto-plugins.rst @@ -2,10 +2,295 @@ Plugin development ================== -API ---- -.. automodule:: libpermian.plugins.api - :noindex: +Overview +======== + +Permian uses a plugin-based architecture that allows you to extend its functionality without modifying the core codebase. Plugins are Python packages that register components through a well-defined API. + +Plugin Structure +================ + +Basic Layout +------------ + +A plugin is a Python package located in ``libpermian/plugins/`` with the following structure:: + + libpermian/plugins/ + └── myplugin/ + ├── __init__.py # Required: Plugin registration code + ├── settings.ini # Optional: Plugin-specific settings + └── [other modules] # Optional: Additional plugin code + +Plugin Loading +-------------- + +Plugins are automatically discovered and loaded when: + +1. They are located in the ``libpermian/plugins/`` directory or path defined in ``PIPELINEPLUGINS_PATH`` +2. They contain an ``__init__.py`` file +3. They are not disabled via ``DISABLED`` file or environment variables. + + Create a ``DISABLED`` file in the plugin directory if you want to disable it by default:: + + touch libpermian/plugins/myplugin/DISABLED + +**Environment Variables:** + +- ``PIPELINEPLUGINS_DISABLE`` - Comma-separated list of plugin names to disable +- ``PIPELINEPLUGINS_ENABLE`` - Comma-separated list of plugin names to enable (overrides DISABLED file) +- ``PIPELINEPLUGINS_PATH`` - Colon-separated paths to additional plugin directories + +Plugin API +========== + +The plugin API is exposed through ``libpermian.plugins.api`` and provides several extension points: + +Events +------ + +Register custom event types that trigger pipeline execution. + +**Register Event Type:** + +.. code-block:: python + + from libpermian.plugins import api + from libpermian.events.base import Event + + @api.events.register('myevent') + class MyEvent(Event): + def __init__(self, settings, type, **kwargs): + super().__init__(settings, type, **kwargs) + + def __str__(self): + return 'MyEvent' + +**Register Event Structure:** + +Register event structures, they provide additional information related to the event + +The structure classes can define special `from_*` classmethods and +`to_*` methods which can be used for conversions either from another +structure or to another structure where the asterisk represents name +under which is the other structure class registered. + +.. code-block:: python + + from libpermian.events.structures.base import BaseStructure + + @api.events.register_structure('mystructure') + class MyStructure(BaseStructure): + def __init__(self, settings, field1, field2=None): + super().__init__(settings) + self.field1 = field1 + self.field2 = field2 + +Workflows +--------- + +Register workflows that define how test cases are executed. + +.. code-block:: python + + from libpermian.workflows.isolated import IsolatedWorkflow + from libpermian.result import Result + + @api.workflows.register("myworkflow") + class MyWorkflow(IsolatedWorkflow): + def setup(self): + """Executed before execute(). Use for preparation tasks.""" + self.reportResult(Result('queued')) + + def execute(self): + """Contains the main test execution logic. Required.""" + self.reportResult(Result('started')) + # Your test logic here + self.log('Log message', 'logfile') + self.addLog('local.txt', '/path/to/file') + + def teardown(self): + """Executed after execute(). Use for cleanup.""" + self.reportResult(Result('complete', 'PASS', final=True)) + + def terminate(self): + """Called asynchronously to stop execution.""" + return False + + def displayStatus(self): + """Return status string for WebUI display.""" + return 'Running my workflow' + +**Grouped Workflows:** + +For workflows that handle multiple test case runs together: + +.. code-block:: python + + from libpermian.workflows.isolated import GroupedWorkflow + + @api.workflows.register("grouped_workflow") + class MyGroupedWorkflow(GroupedWorkflow): + @classmethod + def factory(cls, testRuns, crcList): + """Create instances grouping CRCs as needed.""" + for testcase, crcs in crcList.by_testcase(): + cls(testRuns, crcs) + + def execute(self): + # Use groupReportResult, groupLog, groupAddLog methods + self.groupReportResult(self.crcList, Result('started')) + +Report Senders +-------------- + +Register report senders that handle results reporting to external systems. + +.. code-block:: python + + from libpermian.reportsenders.base import BaseReportSender + + @api.reportsenders.register('myreporter') + class MyReportSender(BaseReportSender): + def processPartialResult(self, result): + """Handle interim results.""" + pass + + def processFinalResult(self, result): + """Handle final test results.""" + pass + + def processTestRunStarted(self): + """Called when test run starts.""" + pass + + def processTestRunFinished(self): + """Called when test run finishes.""" + pass + + def processCaseRunFinished(self, testCaseID): + """Called when a test case run completes.""" + pass + +CLI Commands +------------ + +Register custom CLI commands and arguments. + +**Register Command Parser:** + +.. code-block:: python + + import json + from libpermian.cli.parser import ToPayload + + @api.cli.register_command_parser('mycommand') + def mycommand_parser(base_parser, args): + parser = base_parser + parser.add_argument('id', action=ToPayload, + help='Resource ID') + parser.add_argument('--option', action=ToPayload, + help='Optional parameter') + options = parser.parse_args(args) + return options, json.dumps({ + 'type': options.event_type or 'myevent', + 'mystructure': options.payload + }) + +**Extend Global Arguments:** + +.. code-block:: python + + @api.cli.register_command_args_extension + def extend_args(parser): + parser.add_argument('--my-global-arg', action='store_true') + return parser + +Hooks +----- + +Define and respond to custom hooks for event-driven behavior. + +.. code-block:: python + + # Define a hook + @api.hooks.make + def my_hook(data): + pass + + # Register synchronous callback + @api.hooks.callback_on(my_hook) + def handle_my_hook(data): + # Process the hook + pass + + # Register threaded callback (runs in background) + @api.hooks.threaded_callback_on(my_hook) + def handle_my_hook_async(data): + # Process the hook asynchronously + pass + + # Trigger the hook + my_hook('some data') + +Web UI +------ + +Register Flask blueprints to add custom web UI pages. + +.. code-block:: python + + from flask import Blueprint + + my_blueprint = Blueprint('myplugin', __name__) + + @my_blueprint.route('/mypage') + def my_page(): + return 'Hello from my plugin!' + + @api.webui.register_blueprint(my_blueprint, url_prefix='/myplugin') + +Issue Analyzer +-------------- + +Register issue analyzers for test failure analysis. + +.. code-block:: python + + @api.issueanalyzer.register + class MyAnalyzer: + # Implementation details + pass + +Plugin Settings +=============== + +Plugins can provide default settings via ``settings.ini`` file in the plugin directory: + +.. code-block:: ini + + [mysection] + option1 = value1 + option2 = value2 + +Settings can be accessed through the ``settings`` object passed to plugin components. + +Best Practices +============== + +1. **Naming**: Use descriptive, unique names for your plugin and registered components +2. **Dependencies**: Import from ``libpermian.plugins import api`` at the module level +3. **Error Handling**: Properly handle exceptions and provide meaningful error messages +4. **Logging**: Use the standard Python logging module for debugging +5. **Documentation**: Document your plugin's purpose, configuration options, and usage. This should be done as docstrings in the code. +6. **Testing**: Write unit tests for your plugin in a ``test_*.py`` file Example ------- +See a complete working example in ``libpermian/plugins/example/__init__.py`` which should demonstrates basic plugin API features. + +The ``compose`` plugin is a good example of real Event and Event structure + +API Reference +============= + +For detailed API documentation: :doc:`code/plugins`, :doc:`code/events`, :doc:`code/workflows`, :doc:`code/reportsenders` diff --git a/doc/development/howto-tests.rst b/doc/development/howto-tests.rst index 9246700..595a5d5 100644 --- a/doc/development/howto-tests.rst +++ b/doc/development/howto-tests.rst @@ -2,10 +2,227 @@ Testing ======= ----------- -Unit tests ----------- +Overview +======== ------------------ -Integration tests ------------------ \ No newline at end of file +Permian uses a comprehensive testing strategy with three types of tests: + +1. **Lint tests** - Code quality checks using pylint +2. **Unit tests** - Fast, isolated tests of individual components +3. **Integration tests** - End-to-end tests of complete workflows + +All tests can be run using the Makefile targets inside the container:: + + ./in_container make test # Run all tests + ./in_container make test.lint # Run only lint tests + ./in_container make test.unit # Run only unit tests + ./in_container make test.integration # Run only integration tests + +Test Organization +================= + +Directory Structure +------------------- + +Tests are organized as follows:: + + permian/ + ├── libpermian/ # Source code + │ ├── events/ + │ │ └── test_events.py # Unit tests alongside source + │ ├── plugins/ + │ │ ├── compose/ + │ │ │ └── test_compose.py # Plugin unit tests + │ │ └── test_plugins.py # Plugin system tests + │ └── ... + ├── tests/ # Test-specific files + │ ├── integration/ # Integration test suites + │ │ ├── test_example/ # One directory per test suite + │ │ │ └── test_0.sh # Shell-based integration test + │ │ └── test_testing_plugin/ + │ │ ├── test_1.sh + │ │ └── test_1_result/ # Expected results + │ ├── plugins/ # Test plugin fixtures + │ │ ├── test1_enabled/ + │ │ └── test2_disabled/ + │ ├── test_library/ # Test library for integration tests + │ └── test_settings.ini # Test configuration + ├── run_unit_tests.py # Unit test runner + ├── run_integration_tests.sh # Integration test runner + └── run_pylint.sh # Linter + +Unit Tests +========== + +Unit tests use Python's built-in ``unittest`` framework and test individual components in isolation. + +Writing Unit Tests +------------------ + +Unit test files are named ``test_*.py`` and placed alongside the source code they test. + +**Basic Structure:** + +.. code-block:: python + + import unittest + from unittest.mock import patch, MagicMock + + class TestMyComponent(unittest.TestCase): + def setUp(self): + """Called before each test method.""" + pass + + def tearDown(self): + """Called after each test method.""" + pass + + @classmethod + def setUpClass(cls): + """Called once before all tests in the class.""" + pass + + @classmethod + def tearDownClass(cls): + """Called once after all tests in the class.""" + pass + + def test_basic_functionality(self): + """Test names must start with 'test_'.""" + self.assertEqual(2 + 2, 4) + +Mocking +------- + +Use ``unittest.mock`` for isolating components: + +**Common Mock Patterns:** + +.. code-block:: python + + from unittest.mock import patch, MagicMock, call + + # Mock environment variables + @patch('os.environ', new={'VAR': 'value'}) + def test_with_env(self): + pass + + # Mock function calls + class ResultMock200(): + status_code = 200 + text = 'test' + + @classmethod + def json(cls): + return {'head': {'sha': 'abc123'}} + + @patch('requests.get') + def test_reporting(self, requests_get): + requests_get.return_value = ResultMock200() + +Running Unit Tests +------------------ + +Unit tests are discovered and run using script:: + + ./run_unit_tests.py + +Or inside the container:: + + ./in_container make test.unit + +The test runner: + +1. Discovers all ``test*.py`` in project root and all loaded plugin directories +2. Runs all discovered tests +3. Exits with code 2 if any test fails, debug output written to ``test_debug.log`` + +Integration Tests +================= + +Integration tests verify complete workflows by running the pipeline with real or simulated inputs and comparing output to expected results. + +Writing Integration Tests +------------------------- + +Integration tests are bash scripts that define three functions: ``setup``, ``test``, and ``cleanup``. + +**Basic Structure:** + +.. code-block:: bash + + setup() { + echo "Test description" + # Prepare test environment + TEST_REPORT_DIR=$(mktemp -d) + } + + test() { + # Run the pipeline or specific commands + ./pipeline example -o library.directPath=./tests/test_library + + # Optionally compare results + diff -Naur expected_result $TEST_REPORT_DIR + } + + cleanup() { + # Clean up resources + rm -rf $TEST_REPORT_DIR + } + +Running Integration Tests +-------------------------- + +Integration tests are run using:: + + ./run_integration_tests.sh + +Or inside the container:: + + ./in_container make test.integration + +The runner: + +1. Discovers all ``test_*.sh`` files in ``tests/integration/*/`` +2. For each test: + + - Runs ``setup()`` (exits if setup fails) + - Runs ``test()`` (marks as PASSED or FAILED) + - Runs ``cleanup()`` (reports if cleanup fails) + +3. Exits with: + + - ``0`` - All tests passed + - ``1`` - One or more tests failed + - ``2`` - Setup error + - ``3`` - Cleanup error + +Lint Tests +========== + +Pylint is used to check code quality and catch common errors. + +Running Lint Tests +------------------ + +Linting is run using script:: + + ./run_pylint.sh + +Or inside the container:: + + ./in_container make test.lint + +The linter checks: + +- Core libpermian code +- All loaded plugins (discovered via ``python3 -m libpermian.plugins list --paths``) +- Only errors are reported (``-E`` flag) + +Pylint configuration is in ``.pylintrc`` + +Continuous Integration +====================== + +Tests are designed to run in CI environments. Github Actions is used to run all tests and build documentation on every pull request. +No change should be merged without all tests passing.