diff --git a/qase-behave/README.md b/qase-behave/README.md index de458b26..ea767b69 100644 --- a/qase-behave/README.md +++ b/qase-behave/README.md @@ -38,7 +38,7 @@ For information about attaching files and content or adding comments to test res Qase Behave Reporter supports sending test results to multiple Qase projects simultaneously. You can specify different test case IDs for each project using the `@qase.project_id.PROJECT_CODE:IDS` tag format. -For detailed information and examples, see the [Multi-Project Support](docs/usage.md#multi-project-support) section in the usage documentation. +For detailed information, configuration, and examples, see the [Multi-Project Support Guide](docs/MULTI_PROJECT.md). For example: @@ -71,8 +71,7 @@ Qase Behave Reporter is configured in multiple ways: Environment variables override the values given in the config file, and command line options override both other values. -Configuration options are described in the -[configuration reference](docs/CONFIGURATION.md). +For complete configuration reference, see the [qase-python-commons README](../qase-python-commons/README.md) which contains all available configuration options. ### Example: qase.config.json diff --git a/qase-behave/docs/CONFIGURATION.md b/qase-behave/docs/CONFIGURATION.md deleted file mode 100644 index 82822be8..00000000 --- a/qase-behave/docs/CONFIGURATION.md +++ /dev/null @@ -1,107 +0,0 @@ -# Qase Behave Plugin configuration - -Qase Behave Reporter is configured in multiple ways: - -- using a config file `qase.config.json` -- using environment variables -- using command line options - -Environment variables override the values given in the config file, -and command line options override both other values. - -## Configuration options - -| Description | Config file | Environment variable | CLI option | Default value | Required | Possible values | -|-------------------------------------------|----------------------------|---------------------------------|---------------------------------|-----------------------------------------|----------|----------------------------| -| **Common** | -| Main reporting mode | `mode` | `QASE_MODE` | `qase-mode` | `off` | No | `testops`, `report`, `off` | -| Fallback reporting mode | `fallback` | `QASE_FALLBACK` | `qase-fallback` | `report` | No | `testops`, `report`, `off` | -| Execution plan path | `executionPlan.path` | `QASE_EXECUTION_PLAN_PATH` | `qase-execution-plan-path` | `./build/qase-execution-plan.json` | No | Any string | -| Qase environment | `environment` | `QASE_ENVIRONMENT` | `qase-environment` | `local` | No | Any string | -| Root suite | `rootSuite` | `QASE_ROOT_SUITE` | `qase-root-suite` | | No | Any string | -| Debug logs | `debug` | `QASE_DEBUG` | `qase-debug` | `False` | No | `True`, `False` | -| Exclude parameters from test results | `excludeParams` | `QASE_EXCLUDE_PARAMS` | `qase-exclude-params` | None, don't exclude any parameters | No | Comma-separated list of parameter names | -| **Logging configuration** | -| Enable/disable console logging output | `logging.console` | `QASE_LOGGING_CONSOLE` | `qase-logging-console` | `true` | No | `true`, `false` | -| Enable/disable file logging output | `logging.file` | `QASE_LOGGING_FILE` | `qase-logging-file` | `false` (true when debug=true) | No | `true`, `false` | -| **Qase TestOps mode configuration** | -| Qase project code | `testops.project` | `QASE_TESTOPS_PROJECT` | `qase-testops-project` | | Yes | Any string | -| Qase API token | `testops.api.token` | `QASE_TESTOPS_API_TOKEN` | `qase-testops-api-token` | | Yes | Any string | -| Qase API host. For enterprise users, specify address: `example.qase.io` | `testops.api.host` | `QASE_TESTOPS_API_HOST` | `qase-testops-api-host` | `qase.io` | No | Any string | -| Title of the Qase test run | `testops.run.title` | `QASE_TESTOPS_RUN_TITLE` | `qase-testops-run-title` | `Automated Run {current date and time}` | No | Any string | -| Description of the Qase test run | `testops.run.description` | `QASE_TESTOPS_RUN_DESCRIPTION` | `qase-testops-run-description` | None, leave empty | No | Any string | -| Create test run using a test plan | `testops.plan.id` | `QASE_TESTOPS_PLAN_ID` | `qase-testops-plan-id` | None, don't use plans for the test run | No | Any integer | -| Complete test run after running tests | `testops.run.complete` | `QASE_TESTOPS_RUN_COMPLETE` | `qase-testops-run-complete` | `True` | No | `True`, `False` | -| ID of the Qase test run to report results | `testops.run.id` | `QASE_TESTOPS_RUN_ID` | `qase-testops-run-id` | None, create a new test run | No | Any integer | -| External link type for test run | `testops.run.externalLink.type` | `QASE_TESTOPS_RUN_EXTERNAL_LINK_TYPE` | `qase-testops-run-external-link-type` | None, don't add external link | No | `jiraCloud`, `jiraServer` | -| External link URL for test run | `testops.run.externalLink.link` | `QASE_TESTOPS_RUN_EXTERNAL_LINK_URL` | `qase-testops-run-external-link-url` | None, don't add external link | No | Any string (e.g., "PROJ-1234") | -| Batch size for uploading test results | `testops.batch.size` | `QASE_TESTOPS_BATCH_SIZE` | `qase-testops-batch-size` | 200 | No | 1 to 2000 | -| Create defects in Qase | `testops.defect` | `QASE_TESTOPS_DEFECT` | `qase-testops-defect` | `False`, don't create defects | No | `True`, `False` | -| Test run tags | `testops.run.tags` | `QASE_TESTOPS_RUN_TAGS` | `qase-testops-run-tags` | None, don't add any tags | No | Comma-separated list of tags | -| Test run configurations | `testops.configurations.values` | `QASE_TESTOPS_CONFIGURATIONS_VALUES` | `qase-testops-configurations-values` | None, don't add any configurations | No | Format: "group1=value1,group2=value2" | -| Create configurations if not exists | `testops.configurations.createIfNotExists` | `QASE_TESTOPS_CONFIGURATIONS_CREATE_IF_NOT_EXISTS` | `qase-testops-configurations-create-if-not-exists` | `False`, don't create configurations | No | `True`, `False` | -| Filter results by status | `testops.statusFilter` | `QASE_TESTOPS_STATUS_FILTER` | `qase-testops-status-filter` | None, don't filter any results | No | Comma-separated list of statuses | -| Map test result statuses | `statusMapping` | `STATUS_MAPPING` | `qase-status-mapping` | None, don't map any statuses | No | Format: 'source1=target1,source2=target2' | -| **Qase Report mode configuration** | -| Local path to store report | `report.connection.path` | `QASE_REPORT_CONNECTION_PATH` | `qase-report-connection-path` | `./build/qase-report` | No | Any string | -| Report format | `report.connection.format` | `QASE_REPORT_CONNECTION_FORMAT` | `qase-report-connection-format` | `json` | No | `json`, `jsonp` | -| Driver used for report mode | `report.driver` | `QASE_REPORT_DRIVER` | `qase-report-driver` | `local` | No | `local` | - -## Example `qase.config.json` config: - -```json -{ - "mode": "testops", - "fallback": "report", - "debug": false, - "environment": "local", - "logging": { - "console": true, - "file": false - }, - "report": { - "driver": "local", - "connection": { - "local": { - "path": "./build/qase-report", - "format": "json" - } - } - }, - "testops": { - "api": { - "token": "", - "host": "qase.io" - }, - "run": { - "title": "Regress run", - "description": "Regress run description", - "complete": true, - "tags": ["tag1", "tag2"], - "externalLink": { - "type": "jiraCloud", - "link": "PROJ-1234" - } - }, - "defect": false, - "project": "", - "batch": { - "size": 100 - }, - "statusFilter": ["passed", "failed"], - "configurations": { - "values": [ - { - "name": "group1", - "value": "value1" - }, - { - "name": "group2", - "value": "value2" - } - ], - "createIfNotExists": true - } - } -} -``` diff --git a/qase-behave/docs/MULTI_PROJECT.md b/qase-behave/docs/MULTI_PROJECT.md new file mode 100644 index 00000000..479d7969 --- /dev/null +++ b/qase-behave/docs/MULTI_PROJECT.md @@ -0,0 +1,240 @@ +# Multi-Project Support in Behave + +Qase Behave Reporter supports sending test results to multiple Qase projects simultaneously. This feature allows you to report the same test execution to different projects with different test case IDs, which is useful when: + +* You need to report the same test to different projects +* Different projects track the same functionality with different test case IDs +* You want to maintain separate test runs for different environments or teams + +## Configuration + +For detailed configuration options, refer to the [qase-python-commons README](../../qase-python-commons/README.md#multi-project-support). + +### Basic Multi-Project Configuration + +To enable multi-project support, set the mode to `testops_multi` in your `qase.config.json`: + +```json +{ + "mode": "testops_multi", + "testops": { + "api": { + "token": "", + "host": "qase.io" + }, + "batch": { + "size": 100 + } + }, + "testops_multi": { + "default_project": "PROJ1", + "projects": [ + { + "code": "PROJ1", + "run": { + "title": "PROJ1 Test Run", + "description": "Test run for PROJ1 project", + "complete": true + }, + "environment": "staging" + }, + { + "code": "PROJ2", + "run": { + "title": "PROJ2 Test Run", + "description": "Test run for PROJ2 project", + "complete": true + }, + "environment": "production" + } + ] + } +} +``` + +## Using `@qase.project_id.PROJECT_CODE:IDS` Tags + +In Behave, you use tags to specify multi-project mappings. The format is `@qase.project_id.PROJECT_CODE:IDS` where `PROJECT_CODE` is the project code and `IDS` is a comma-separated list of test case IDs. + +### Basic Usage + +```gherkin +Feature: Example tests + + # Single project with single ID + @qase.project_id.PROJ1:123 + Scenario: Example test with single ID + Given I have a simple test + When I run it + Then it should pass + + # Single project with multiple IDs + @qase.project_id.PROJ1:123,124 + Scenario: Example test with multiple IDs + Given I have a simple test + When I run it + Then it should pass +``` + +### Multiple Projects + +You can add multiple `@qase.project_id` tags to send results to multiple projects: + +```gherkin +Feature: Example tests + + # Multiple projects, each with single ID + @qase.project_id.PROJ1:123 + @qase.project_id.PROJ2:456 + Scenario: Example test for multiple projects + Given I have a simple test + When I run it + Then it should pass + + # Multiple projects, each with multiple IDs + @qase.project_id.PROJ1:123,124 + @qase.project_id.PROJ2:456,457 + Scenario: Complex multi-project test + Given I have a simple test + When I run it + Then it should pass +``` + +### Combining with Other Tags + +You can combine `@qase.project_id` tags with other Qase tags: + +```gherkin +Feature: User authentication + + @qase.project_id.PROJ1:123 + @qase.fields:{"description":"User_login_test","severity":"critical"} + @qase.suite:Authentication + Scenario: User login test + Given I am on the login page + When I enter valid credentials + Then I should be logged in +``` + +### Scenarios Without Project Mapping + +If a scenario doesn't have any `@qase.project_id` tags, it will be sent to the `default_project` specified in your configuration: + +```gherkin +Feature: Example tests + + Scenario: Test without project ID + Given I have a simple test + When I run it + Then it should pass +``` + +If `default_project` is not specified, the first project from the `projects` array will be used. + +## Feature-Level Tags + +You can apply project ID tags at the Feature level, and all scenarios in that feature will inherit the mapping: + +```gherkin +@qase.project_id.PROJ1:100 +Feature: User management + + Scenario: Create user + Given I am logged in + When I create a new user + Then the user should be created + + Scenario: Delete user + Given I am logged in + When I delete a user + Then the user should be deleted +``` + +You can override the feature-level mapping for specific scenarios: + +```gherkin +@qase.project_id.PROJ1:100 +Feature: User management + + Scenario: Create user + Given I am logged in + When I create a new user + Then the user should be created + + @qase.project_id.PROJ2:200 + Scenario: Delete user + Given I am logged in + When I delete a user + Then the user should be deleted +``` + +## Tag Format Details + +The tag format is case-sensitive and follows this pattern: + +* Format: `@qase.project_id.PROJECT_CODE:ID1,ID2,ID3` +* `PROJECT_CODE`: Must match exactly with the project code in your configuration +* `IDS`: Comma-separated list of integers (no spaces around commas) +* Multiple tags can be used for multiple projects + +Examples: + +```gherkin +# Valid formats +@qase.project_id.PROJ1:123 +@qase.project_id.PROJ1:123,124,125 +@qase.project_id.PROJ1:123 @qase.project_id.PROJ2:456 + +# Invalid formats (will be ignored) +@qase.project_id.proj1:123 # Wrong case +@qase.project_id.PROJ1: 123 # Space after colon +@qase.project_id.PROJ1:123, 124 # Space in ID list +``` + +## Important Notes + +1. **Project Codes Must Match**: The project codes used in tags must exactly match the codes specified in your `testops_multi.projects` configuration. + +2. **Test Case IDs**: Each project can have different test case IDs for the same scenario. This allows you to maintain separate test case tracking in different projects. + +3. **Test Run Creation**: Each project will have its own test run created (or use an existing run if `run.id` is specified in the project configuration). + +4. **Results Distribution**: Test results are sent to all specified projects simultaneously. If a scenario fails, the failure will be reported to all projects. + +5. **Default Project**: Scenarios without explicit project mapping will be sent to the `default_project`. If no `default_project` is specified, the first project in the configuration will be used. + +6. **Mode Requirement**: You must set `mode` to `testops_multi` in your configuration file. Using `testops` mode will not work with `@qase.project_id` tags. + +7. **Tag Parsing**: Tags are case-insensitive for the `qase.project_id` part, but the project code itself is case-sensitive. + +## Examples + +See the [multi-project examples](../../../examples/multiproject/behave/) directory for complete working examples. + +## Troubleshooting + +### Test results not appearing in projects + +* Verify that `mode` is set to `testops_multi` in your `qase.config.json` +* Check that project codes in tags match exactly (case-sensitive) with configuration +* Ensure all projects are properly configured in `testops_multi.projects` +* Check debug logs for any errors during test run creation +* Verify tag format is correct (no spaces, proper comma separation) + +### Scenarios sent to wrong project + +* Verify the `default_project` setting if scenarios don't have explicit project mapping +* Check that project codes are case-sensitive and match exactly +* Review feature-level tags that might be affecting scenario mapping + +### Multiple test runs created + +* This is expected behavior - each project gets its own test run +* To use existing runs, specify `run.id` in each project's configuration + +### Tag not recognized + +* Ensure the tag starts with `@qase.project_id.` (case-insensitive) +* Check that there are no spaces in the tag +* Verify the colon (`:`) is used correctly to separate project code from IDs +* Ensure IDs are comma-separated without spaces diff --git a/qase-behave/docs/usage.md b/qase-behave/docs/usage.md index 6bd9f732..bac4d509 100644 --- a/qase-behave/docs/usage.md +++ b/qase-behave/docs/usage.md @@ -3,6 +3,8 @@ This guide demonstrates how to integrate Qase with Behave, providing instructions on how to add Qase IDs, fields and suites to your test cases. +> **Configuration:** For complete configuration reference including all available options, environment variables, and examples, see the [qase-python-commons README](../../qase-python-commons/README.md). + --- ## Adding QaseID to a Test @@ -24,36 +26,9 @@ Feature: Example tests ### Multi-Project Support -To send test results to multiple projects with different test case IDs, use the `@qase.project_id.PROJECT_CODE:IDS` tag: - -```gherkin -Feature: Example tests - - # Single project with multiple IDs - @qase.project_id.PROJ1:123,124 - Scenario: Example test with multiple IDs - Given I have a simple test - When I run it - Then it should pass - - # Multiple projects with different IDs - @qase.project_id.PROJ1:123 - @qase.project_id.PROJ2:456 - Scenario: Example test for multiple projects - Given I have a simple test - When I run it - Then it should pass - - # Multiple projects with multiple IDs each - @qase.project_id.PROJ1:123,124 - @qase.project_id.PROJ2:456,457 - Scenario: Complex multi-project test - Given I have a simple test - When I run it - Then it should pass -``` +Qase Behave Reporter supports sending test results to multiple Qase projects simultaneously with different test case IDs for each project. -**Note:** When using `@qase.project_id`, the test results will be sent to all specified projects. Make sure to configure `testops_multi` mode in your `qase.config.json` file. +For detailed information, configuration, examples, and troubleshooting, see the [Multi-Project Support Guide](MULTI_PROJECT.md). --- diff --git a/qase-pytest/README.md b/qase-pytest/README.md index ceddc32f..2e88d51f 100644 --- a/qase-pytest/README.md +++ b/qase-pytest/README.md @@ -26,8 +26,8 @@ Qase Pytest Reporter is configured in multiple ways: Environment variables override the values given in the config file, and command line options override both other values. -Configuration options are described in the -[configuration reference](docs/CONFIGURATION.md). +For complete configuration reference, see the [qase-python-commons README](../qase-python-commons/README.md) which contains all available configuration options. + ### Example: qase.config.json @@ -80,7 +80,7 @@ For detailed instructions on using annotations and methods, refer to [Usage](doc Qase Pytest Reporter supports sending test results to multiple Qase projects simultaneously. You can specify different test case IDs for each project using the `@qase.project_id()` decorator. -For detailed information and examples, see the [Multi-Project Support](docs/usage.md#multi-project-support) section in the usage documentation. +For detailed information, configuration, and examples, see the [Multi-Project Support Guide](docs/MULTI_PROJECT.md). ### Link tests with test cases in Qase TestOps diff --git a/qase-pytest/docs/CONFIGURATION.md b/qase-pytest/docs/CONFIGURATION.md deleted file mode 100644 index aab27531..00000000 --- a/qase-pytest/docs/CONFIGURATION.md +++ /dev/null @@ -1,116 +0,0 @@ -# Qase Pytest Plugin configuration - -Qase Pytest Reporter is configured in multiple ways: - -- using a config file `qase.config.json` -- using environment variables -- using command line options - -Environment variables override the values given in the config file, -and command line options override both other values. - -## Configuration options - -| Description | Config file | Environment variable | CLI option | Default value | Required | Possible values | -|------------------------------------------------|--------------------------------------|----------------------------------|------------------------------------|-----------------------------------------|----------|----------------------------| -| **Common** | -| Main reporting mode | `mode` | `QASE_MODE` | `--qase-mode` | `off` | No | `testops`, `report`, `off` | -| Fallback reporting mode | `fallback` | `QASE_FALLBACK` | `--qase-fallback` | `report` | No | `testops`, `report`, `off` | -| Execution plan path | `executionPlan.path` | `QASE_EXECUTION_PLAN_PATH` | `--qase-execution-plan-path` | `./build/qase-execution-plan.json` | No | Any string | -| Qase environment | `environment` | `QASE_ENVIRONMENT` | `--qase-environment` | `local` | No | Any string | -| Root suite | `rootSuite` | `QASE_ROOT_SUITE` | `--qase-root-suite` | | No | Any string | -| Debug logs | `debug` | `QASE_DEBUG` | `--qase-debug` | false | No | `true`, `false` | -| Exclude parameters from test results | `excludeParams` | `QASE_EXCLUDE_PARAMS` | `--qase-exclude-params` | None, don't exclude any parameters | No | Comma-separated list of parameter names | -| **Logging configuration** | -| Enable/disable console logging output | `logging.console` | `QASE_LOGGING_CONSOLE` | `--qase-logging-console` | `true` | No | `true`, `false` | -| Enable/disable file logging output | `logging.file` | `QASE_LOGGING_FILE` | `--qase-logging-file` | `false` (true when debug=true) | No | `true`, `false` | -| **Qase TestOps mode configuration** | -| Qase project code | `testops.project` | `QASE_TESTOPS_PROJECT` | `--qase-testops-project` | | Yes | Any string | -| Qase API token | `testops.api.token` | `QASE_TESTOPS_API_TOKEN` | `--qase-testops-api-token` | | Yes | Any string | -| Qase API host. For enterprise users, specify address: `example.qase.io` | `testops.api.host` | `QASE_TESTOPS_API_HOST` | `--qase-testops-api-host` | `qase.io` | No | Any string | -| Title of the Qase test run | `testops.run.title` | `QASE_TESTOPS_RUN_TITLE` | `--qase-testops-run-title` | `Automated Run {current date and time}` | No | Any string | -| Description of the Qase test run | `testops.run.description` | `QASE_TESTOPS_RUN_DESCRIPTION` | `--qase-testops-run-description` | None, leave empty | No | Any string | -| Create test run using a test plan | `testops.plan.id` | `QASE_TESTOPS_PLAN_ID` | `--qase-testops-plan-id` | None, don't use plans for the test run | No | Any integer | -| Complete test run after running tests | `testops.run.complete` | `QASE_TESTOPS_RUN_COMPLETE` | `--qase-testops-run-complete` | `True` | No | `true`, `false` | -| ID of the Qase test run to report results | `testops.run.id` | `QASE_TESTOPS_RUN_ID` | `--qase-testops-run-id` | None, create a new test run | No | Any integer | -| Batch size for uploading test results | `testops.batch.size` | `QASE_TESTOPS_BATCH_SIZE` | `--qase-testops-batch-size` | 200 | No | 1 to 2000 | -| Create defects in Qase | `testops.defect` | `QASE_TESTOPS_DEFECT` | `--qase-testops-defect` | `False`, don't create defects | No | `True`, `False` | -| Test run tags | `testops.run.tags` | `QASE_TESTOPS_RUN_TAGS` | `--qase-testops-run-tags` | None, don't add any tags | No | Comma-separated list of tags | -| External link type for test run | `testops.run.externalLink.type` | `QASE_TESTOPS_RUN_EXTERNAL_LINK_TYPE` | `--qase-testops-run-external-link-type` | None, don't add external link | No | `jiraCloud`, `jiraServer` | -| External link URL for test run | `testops.run.externalLink.link` | `QASE_TESTOPS_RUN_EXTERNAL_LINK_URL` | `--qase-testops-run-external-link-url` | None, don't add external link | No | Any string (e.g., "PROJ-1234") | -| Test run configurations | `testops.configurations.values` | `QASE_TESTOPS_CONFIGURATIONS_VALUES` | `--qase-testops-configurations-values` | None, don't add any configurations | No | Format: "group1=value1,group2=value2" | -| Create configurations if not exists | `testops.configurations.createIfNotExists` | `QASE_TESTOPS_CONFIGURATIONS_CREATE_IF_NOT_EXISTS` | `--qase-testops-configurations-create-if-not-exists` | `False`, don't create configurations | No | `True`, `False` | -| Filter results by status | `testops.statusFilter` | `QASE_TESTOPS_STATUS_FILTER` | `--qase-testops-status-filter` | None, don't filter any results | No | Comma-separated list of statuses | -| Map test result statuses | `statusMapping` | `STATUS_MAPPING` | `--qase-status-mapping` | None, don't map any statuses | No | Format: 'source1=target1,source2=target2' | -| **Qase Report mode configuration** | -| Local path to store report | `report.connection.path` | `QASE_REPORT_CONNECTION_PATH` | `--qase-report-connection-path` | `./build/qase-report` | No | Any string | -| Report format | `report.connection.format` | `QASE_REPORT_CONNECTION_FORMAT` | `--qase-report-connection-format` | `json` | No | `json`, `jsonp` | -| Driver used for report mode | `report.driver` | `QASE_REPORT_DRIVER` | `--qase-report-driver` | `local` | No | `local` | -| **Framework specific options** | -| **Pytest** | -| Capture logs | `framework.pytest.captureLogs` | `QASE_PYTEST_CAPTURE_LOGS` | `--qase-pytest-capture-logs` | `False` | No | `true`, `false` | -| XFail status for failed tests | `framework.pytest.xfailStatus.xfail` | `QASE_PYTEST_XFAIL_STATUS_XFAIL` | `--qase-pytest-xfail-status-xfail` | `Skipped` | No | Any string | -| XFail status for passed tests | `framework.pytest.xfailStatus.xpass` | `QASE_PYTEST_XFAIL_STATUS_XPASS` | `--qase-pytest-xfail-status-xpass` | `Passed` | No | Any string | -| **Earlier versions** | -| **qase-pytest v5.x** | -| TestOps bulk (always on since v6) | `testops.bulk` | `QASE_TESTOPS_BULK` | `--qase-testops-bulk` | `True` | No | `true`, `false` | -| Execution chunk size (changed to `batch.size`) | `testops.chunk` | `QASE_TESTOPS_CHUNK` | `--qase-testops-chunk` | 200 | No | 1 to 2000 | - -## Example `qase.config.json` config: - -```json -{ - "mode": "testops", - "fallback": "report", - "debug": false, - "environment": "local", - "logging": { - "console": true, - "file": false - }, - "report": { - "driver": "local", - "connection": { - "local": { - "path": "./build/qase-report", - "format": "json" - } - } - }, - "testops": { - "api": { - "token": "", - "host": "qase.io" - }, - "run": { - "title": "Regress run", - "description": "Regress run description", - "complete": true, - "tags": ["tag1", "tag2"], - "externalLink": { - "type": "jiraCloud", - "link": "PROJ-1234" - } - }, - "defect": false, - "project": "", - "batch": { - "size": 100 - }, - "statusFilter": ["passed", "failed"], - "configurations": { - "values": [ - { - "name": "group1", - "value": "value1" - }, - { - "name": "group2", - "value": "value2" - } - ], - "createIfNotExists": true - } - } -} -``` diff --git a/qase-pytest/docs/MULTI_PROJECT.md b/qase-pytest/docs/MULTI_PROJECT.md new file mode 100644 index 00000000..47d04907 --- /dev/null +++ b/qase-pytest/docs/MULTI_PROJECT.md @@ -0,0 +1,206 @@ +# Multi-Project Support in Pytest + +Qase Pytest Reporter supports sending test results to multiple Qase projects simultaneously. This feature allows you to report the same test execution to different projects with different test case IDs, which is useful when: + +* You need to report the same test to different projects +* Different projects track the same functionality with different test case IDs +* You want to maintain separate test runs for different environments or teams + +## Configuration + +For detailed configuration options, refer to the [qase-python-commons README](../../qase-python-commons/README.md#multi-project-support). + +### Basic Multi-Project Configuration + +To enable multi-project support, set the mode to `testops_multi` in your `qase.config.json`: + +```json +{ + "mode": "testops_multi", + "testops": { + "api": { + "token": "", + "host": "qase.io" + }, + "batch": { + "size": 100 + } + }, + "testops_multi": { + "default_project": "PROJ1", + "projects": [ + { + "code": "PROJ1", + "run": { + "title": "PROJ1 Test Run", + "description": "Test run for PROJ1 project", + "complete": true + }, + "environment": "staging" + }, + { + "code": "PROJ2", + "run": { + "title": "PROJ2 Test Run", + "description": "Test run for PROJ2 project", + "complete": true + }, + "environment": "production" + } + ] + } +} +``` + +## Using `@qase.project_id()` Decorator + +The `@qase.project_id()` decorator allows you to specify which project(s) a test should be reported to and which test case IDs to use for each project. + +### Basic Usage + +```python +from qase.pytest import qase + +# Single project with single ID +@qase.project_id("PROJ1", 123) +def test_example(): + assert True + +# Single project with multiple IDs +@qase.project_id("PROJ1", [123, 124]) +def test_multiple_ids(): + assert True +``` + +### Multiple Projects + +You can apply multiple `@qase.project_id()` decorators to send results to multiple projects: + +```python +from qase.pytest import qase + +# Multiple projects, each with single ID +@qase.project_id("PROJ1", 123) +@qase.project_id("PROJ2", 456) +def test_multiple_projects(): + assert True + +# Multiple projects, each with multiple IDs +@qase.project_id("PROJ1", [123, 124]) +@qase.project_id("PROJ2", [456, 457]) +def test_complex_multi_project(): + assert True +``` + +### Combining with Other Decorators + +You can combine `@qase.project_id()` with other Qase decorators: + +```python +from qase.pytest import qase + +@qase.project_id("PROJ1", 123) +@qase.title("User Login Test") +@qase.fields( + ("severity", "critical"), + ("priority", "high") +) +def test_login(): + assert True +``` + +### Tests Without Project Mapping + +If a test doesn't have any `@qase.project_id()` decorator, it will be sent to the `default_project` specified in your configuration: + +```python +def test_without_id(): + """This test will be sent to the default_project from config.""" + assert True +``` + +If `default_project` is not specified, the first project from the `projects` array will be used. + +## Parametrized Tests + +Multi-project support works with parametrized tests: + +```python +import pytest +from qase.pytest import qase + +@pytest.mark.parametrize("value", [1, 2, 3]) +@qase.project_id("PROJ1", 8) +def test_parametrized(value): + assert value > 0 +``` + +Each parametrized test instance will be reported to the specified project with the same test case ID. + +## Test Classes + +You can apply `@qase.project_id()` to test classes: + +```python +from qase.pytest import qase + +@qase.project_id("PROJ1", 100) +class TestUserManagement: + def test_create_user(self): + assert True + + def test_delete_user(self): + assert True +``` + +All test methods in the class will inherit the project mapping. You can override it for specific methods: + +```python +from qase.pytest import qase + +@qase.project_id("PROJ1", 100) +class TestUserManagement: + def test_create_user(self): + assert True + + @qase.project_id("PROJ2", 200) + def test_delete_user(self): + assert True +``` + +## Important Notes + +1. **Project Codes Must Match**: The project codes used in `@qase.project_id()` must exactly match the codes specified in your `testops_multi.projects` configuration. + +2. **Test Case IDs**: Each project can have different test case IDs for the same test. This allows you to maintain separate test case tracking in different projects. + +3. **Test Run Creation**: Each project will have its own test run created (or use an existing run if `run.id` is specified in the project configuration). + +4. **Results Distribution**: Test results are sent to all specified projects simultaneously. If a test fails, the failure will be reported to all projects. + +5. **Default Project**: Tests without explicit project mapping will be sent to the `default_project`. If no `default_project` is specified, the first project in the configuration will be used. + +6. **Mode Requirement**: You must set `mode` to `testops_multi` in your configuration file. Using `testops` mode will not work with `@qase.project_id()` decorators. + +## Examples + +See the [multi-project examples](../../../examples/multiproject/pytest/) directory for complete working examples. + +## Troubleshooting + +### Test results not appearing in projects + +* Verify that `mode` is set to `testops_multi` in your `qase.config.json` +* Check that project codes in decorators match exactly with configuration +* Ensure all projects are properly configured in `testops_multi.projects` +* Check debug logs for any errors during test run creation + +### Tests sent to wrong project + +* Verify the `default_project` setting if tests don't have explicit project mapping +* Check that project codes are case-sensitive and match exactly + +### Multiple test runs created + +* This is expected behavior - each project gets its own test run +* To use existing runs, specify `run.id` in each project's configuration diff --git a/qase-pytest/docs/usage.md b/qase-pytest/docs/usage.md index 748f24b6..fa73e1b9 100644 --- a/qase-pytest/docs/usage.md +++ b/qase-pytest/docs/usage.md @@ -2,6 +2,8 @@ This guide demonstrates how to integrate Qase with Pytest, providing instructions on how to add Qase IDs, fields, suites, and other metadata to your test cases. +> **Configuration:** For complete configuration reference including all available options, environment variables, and examples, see the [qase-python-commons README](../../qase-python-commons/README.md). + --- ## Adding QaseID to a Test @@ -22,30 +24,9 @@ def test_multiple_ids(): ### Multi-Project Support -To send test results to multiple projects with different test case IDs, use the `@qase.project_id` decorator: - -```python -from qase.pytest import qase - -# Single project with multiple IDs -@qase.project_id("PROJ1", [123, 124]) -def test_example(): - assert True - -# Multiple projects with different IDs -@qase.project_id("PROJ1", 123) -@qase.project_id("PROJ2", 456) -def test_multiple_projects(): - assert True +Qase Pytest Reporter supports sending test results to multiple Qase projects simultaneously with different test case IDs for each project. -# Multiple projects with multiple IDs each -@qase.project_id("PROJ1", [123, 124]) -@qase.project_id("PROJ2", [456, 457]) -def test_complex_multi_project(): - assert True -``` - -**Note:** When using `@qase.project_id`, the test results will be sent to all specified projects. Make sure to configure `testops_multi` mode in your `qase.config.json` file. +For detailed information, configuration, examples, and troubleshooting, see the [Multi-Project Support Guide](MULTI_PROJECT.md). --- @@ -225,75 +206,14 @@ def test_combined(browser, test_data): ## Advanced Configuration -### Profilers - -Enable profilers to collect additional data: - -#### Sleep Profiler - -Sleep profiler is a special profiler that allows you to measure the time taken by the test. Each sleep will be reported as a separate step with the duration of the sleep. - -```bash -pytest --qase-profilers=sleep -``` - -```python -from qase.pytest import qase - -def test_example(): - time.sleep(1) - assert True -``` - -#### Network Profiler - -Network profiler is a special profiler that allows you to measure the time taken by the network requests. Each network request will be reported as a separate step with the request details. - -```bash -pytest --qase-profilers=network -``` - -```python -import requests - -def test_example(): - requests.get("https://api.qase.io/v1/projects") - assert True -``` - ---- - -### Log Capture - -Capture pytest logs. All logs will be reported as a attachment to the test case. - -```bash -pytest --qase-pytest-capture-logs=true -``` - -### XFail Status - -Configure xfail status mapping: - -```bash -pytest --qase-pytest-xfail-status-xfail=skipped -``` - ---- - -## Execution Plans +For complete configuration options including profilers, log capture, xfail status, execution plans, and all other settings, see the [qase-python-commons README](../../qase-python-commons/README.md) and [Pytest Configuration Reference](CONFIGURATION.md). -You can use execution plans to run only specific tests: +### Quick Reference -```bash -pytest --qase-execution-plan-path=plan.json -``` - -The execution plan file should contain a list of Qase IDs: - -```json -[1, 2, 3, 4, 5] -``` +* **Profilers**: Use `--qase-profilers=sleep,network` to enable profilers +* **Log Capture**: Use `--qase-pytest-capture-logs=true` to capture pytest logs +* **XFail Status**: Use `--qase-pytest-xfail-status-xfail=skipped` to configure xfail status +* **Execution Plans**: Use `--qase-execution-plan-path=plan.json` to run specific tests --- diff --git a/qase-python-commons/README.md b/qase-python-commons/README.md index 7fda093d..a657d5dc 100644 --- a/qase-python-commons/README.md +++ b/qase-python-commons/README.md @@ -2,13 +2,426 @@ ## Description -This package contains reporters for Qase TestOps and Qase Report that are used in following projects: +This module is an SDK for developing test reporters for Qase TMS. +It's using `qase-api-client` as an API client, and all Qase Python reporters are, in turn, +using this package. +You should use it if you're developing your own test reporter for a special-purpose framework. -- [qase-pytest](https://github.com/qase-tms/qase-python/tree/main/qase-pytest) -- [qase-robotframework](https://github.com/qase-tms/qase-python/tree/main/qase-robotframework) -- [qase-behave](https://github.com/qase-tms/qase-python/tree/main/qase-behave) -- [qase-tavern](https://github.com/qase-tms/qase-python/tree/main/qase-tavern) +To report results from tests using a popular framework or test runner, +don't install this module directly and +use the corresponding reporter module instead: -## How to install +* [Pytest](https://github.com/qase-tms/qase-python/tree/main/qase-pytest#readme) +* [Behave](https://github.com/qase-tms/qase-python/tree/main/qase-behave#readme) +* [Robot Framework](https://github.com/qase-tms/qase-python/tree/main/qase-robotframework#readme) +* [Tavern](https://github.com/qase-tms/qase-python/tree/main/qase-tavern#readme) -`pip install qase-python-commons` +## Installation + +```bash +pip install qase-python-commons +``` + +## Configuration + +Qase Python Reporters can be configured in multiple ways: + +* using a config file `qase.config.json` +* using environment variables +* using command line options (for frameworks that support it, like pytest and tavern) + +Environment variables override the values given in the config file, +and command line options override both other values. + +All configuration options are listed in the tables below: + +### Common Configuration + +| Description | Config file | Environment variable | Default value | Required | Possible values | +|-----------------------------------------------------------------------------------------------------------------------|----------------------------|---------------------------------|-----------------------------------------|----------|----------------------------| +| **Common** | | | | | | +| Mode of reporter | `mode` | `QASE_MODE` | `off` | No | `testops`, `testops_multi`, `report`, `off` | +| Fallback mode of reporter | `fallback` | `QASE_FALLBACK` | `off` | No | `testops`, `testops_multi`, `report`, `off` | +| Environment | `environment` | `QASE_ENVIRONMENT` | undefined | No | Any string | +| Root suite | `rootSuite` | `QASE_ROOT_SUITE` | undefined | No | Any string | +| Enable debug logs | `debug` | `QASE_DEBUG` | `False` | No | `True`, `False` | +| Execution plan path | `executionPlan.path` | `QASE_EXECUTION_PLAN_PATH` | `./build/qase-execution-plan.json` | No | Any string | +| Exclude parameters from test results | `excludeParams` | `QASE_EXCLUDE_PARAMS` | undefined | No | Comma-separated list of parameter names | +| Map test result statuses to different values (format: `fromStatus=toStatus`) | `statusMapping` | `QASE_STATUS_MAPPING` | undefined | No | Object mapping statuses (e.g., `{"invalid": "failed", "skipped": "passed"}`) | +| **Logging configuration** | | | | | | +| Enable/disable console output for reporter logs | `logging.console` | `QASE_LOGGING_CONSOLE` | `True` | No | `True`, `False` | +| Enable/disable file output for reporter logs | `logging.file` | `QASE_LOGGING_FILE` | Same as `debug` setting | No | `True`, `False` | +| **Qase Report configuration** | | | | | | +| Driver used for report mode | `report.driver` | `QASE_REPORT_DRIVER` | `local` | No | `local` | +| Path to save the report | `report.connection.path` | `QASE_REPORT_CONNECTION_PATH` | `./build/qase-report` | No | Any string | +| Local report format | `report.connection.format` | `QASE_REPORT_CONNECTION_FORMAT` | `json` | No | `json`, `jsonp` | +| **Qase TestOps configuration (single project)** | | | | | | +| Token for [API access](https://developers.qase.io/#authentication) | `testops.api.token` | `QASE_TESTOPS_API_TOKEN` | undefined | Yes* | Any string | +| Qase API host. For enterprise users, specify address: `example.qase.io` | `testops.api.host` | `QASE_TESTOPS_API_HOST` | `qase.io` | No | Any string | +| Code of your project, which you can take from the URL: `https://app.qase.io/project/DEMOTR` - `DEMOTR` is the project code | `testops.project` | `QASE_TESTOPS_PROJECT` | undefined | Yes* | Any string | +| Qase test run ID | `testops.run.id` | `QASE_TESTOPS_RUN_ID` | undefined | No | Any integer | +| Qase test run title | `testops.run.title` | `QASE_TESTOPS_RUN_TITLE` | `Automated run ` | No | Any string | +| Qase test run description | `testops.run.description` | `QASE_TESTOPS_RUN_DESCRIPTION` | ` automated run` | No | Any string | +| Qase test run complete | `testops.run.complete` | `QASE_TESTOPS_RUN_COMPLETE` | `True` | No | `True`, `False` | +| Array of tags to be added to the test run | `testops.run.tags` | `QASE_TESTOPS_RUN_TAGS` | `[]` | No | Array of strings | +| External link to associate with test run (e.g., Jira ticket) | `testops.run.externalLink` | `QASE_TESTOPS_RUN_EXTERNAL_LINK` | undefined | No | JSON object with `type` (`jiraCloud` or `jiraServer`) and `link` (e.g., `PROJ-123`) | +| Qase test plan ID | `testops.plan.id` | `QASE_TESTOPS_PLAN_ID` | undefined | No | Any integer | +| Size of batch for sending test results | `testops.batch.size` | `QASE_TESTOPS_BATCH_SIZE` | `200` | No | Any integer (1 to 2000) | +| Enable defects for failed test cases | `testops.defect` | `QASE_TESTOPS_DEFECT` | `False` | No | `True`, `False` | +| Filter test results by status (comma-separated list of statuses to exclude from reporting) | `testops.statusFilter` | `QASE_TESTOPS_STATUS_FILTER` | undefined | No | Array of strings (`passed`, `failed`, `skipped`, `invalid`) | +| Configuration values to create/find in groups (format: `group1=value1,group2=value2`) | `testops.configurations.values` | `QASE_TESTOPS_CONFIGURATIONS_VALUES` | undefined | No | Array of objects with `name` and `value` fields | +| Create configuration groups if they don't exist | `testops.configurations.createIfNotExists` | `QASE_TESTOPS_CONFIGURATIONS_CREATE_IF_NOT_EXISTS` | `false` | No | `True`, `False` | +| Enable public report link generation and display after test run completion | `testops.showPublicReportLink` | `QASE_TESTOPS_SHOW_PUBLIC_REPORT_LINK` | `False` | No | `True`, `False` | +| **Qase TestOps Multi-Project configuration** | | | | | | +| Default project code for tests without explicit project mapping | `testops_multi.default_project` | `QASE_TESTOPS_MULTI_DEFAULT_PROJECT` | undefined | No | Any string (must match one of the project codes in `projects`) | +| Array of project configurations | `testops_multi.projects` | N/A (use config file) | `[]` | Yes** | Array of project configuration objects | +| Project code | `testops_multi.projects[].code` | N/A | undefined | Yes** | Any string | +| Project-specific test run title | `testops_multi.projects[].run.title` | N/A | `Automated Run ` | No | Any string | +| Project-specific test run description | `testops_multi.projects[].run.description` | N/A | `Automated Run ` | No | Any string | +| Project-specific test run complete | `testops_multi.projects[].run.complete` | N/A | `True` | No | `True`, `False` | +| Project-specific test run ID | `testops_multi.projects[].run.id` | N/A | undefined | No | Any integer | +| Project-specific test run tags | `testops_multi.projects[].run.tags` | N/A | `[]` | No | Array of strings | +| Project-specific external link | `testops_multi.projects[].run.externalLink` | N/A | undefined | No | JSON object with `type` and `link` | +| Project-specific test plan ID | `testops_multi.projects[].plan.id` | N/A | undefined | No | Any integer | +| Project-specific environment | `testops_multi.projects[].environment` | N/A | Uses global `environment` if not set | No | Any string or integer (environment ID) | + +\* Required when using `testops` mode +\** Required when using `testops_multi` mode + +### Framework-Specific Configuration + +#### Pytest + +| Description | Config file | Environment variable | CLI option | Default value | Required | Possible values | +|------------------------------------------------|--------------------------------------|----------------------------------|------------------------------------|-----------------------------------------|----------|----------------------------| +| Capture logs | `framework.pytest.captureLogs` | `QASE_PYTEST_CAPTURE_LOGS` | `--qase-pytest-capture-logs` | `False` | No | `true`, `false` | +| XFail status for failed tests | `framework.pytest.xfailStatus.xfail` | `QASE_PYTEST_XFAIL_STATUS_XFAIL` | `--qase-pytest-xfail-status-xfail` | `Skipped` | No | Any string | +| XFail status for passed tests | `framework.pytest.xfailStatus.xpass` | `QASE_PYTEST_XFAIL_STATUS_XPASS` | `--qase-pytest-xfail-status-xpass` | `Passed` | No | Any string | + +#### Behave + +Behave reporter uses the same common configuration options. There are no framework-specific options for Behave. + +#### Robot Framework + +Robot Framework reporter uses the same common configuration options. There are no framework-specific options for Robot Framework. + +#### Tavern + +Tavern reporter uses the same common configuration options. There are no framework-specific options for Tavern. + +## Configuration Examples + +### Single Project Configuration (`testops` mode) + +Example `qase.config.json` config: + +```json +{ + "mode": "testops", + "fallback": "report", + "debug": false, + "environment": "local", + "excludeParams": ["password", "token"], + "statusMapping": { + "invalid": "failed", + "skipped": "passed" + }, + "logging": { + "console": true, + "file": true + }, + "report": { + "driver": "local", + "connection": { + "local": { + "path": "./build/qase-report", + "format": "json" + } + } + }, + "testops": { + "api": { + "token": "", + "host": "qase.io" + }, + "project": "", + "run": { + "title": "Regress run", + "description": "Regress run description", + "complete": true, + "tags": ["tag1", "tag2"], + "externalLink": { + "type": "jiraCloud", + "link": "PROJ-123" + } + }, + "defect": false, + "batch": { + "size": 100 + }, + "statusFilter": ["passed", "skipped"], + "showPublicReportLink": true, + "configurations": { + "values": [ + { + "name": "group1", + "value": "value1" + }, + { + "name": "group2", + "value": "value2" + } + ], + "createIfNotExists": true + } + }, + "framework": { + "pytest": { + "captureLogs": true, + "xfailStatus": { + "xfail": "Skipped", + "xpass": "Passed" + } + } + } +} +``` + +### Multi-Project Configuration (`testops_multi` mode) + +Example `qase.config.json` config for multi-project reporting: + +```json +{ + "mode": "testops_multi", + "fallback": "report", + "debug": false, + "environment": "local", + "logging": { + "console": true, + "file": false + }, + "report": { + "driver": "local", + "connection": { + "local": { + "path": "./build/qase-report", + "format": "json" + } + } + }, + "testops": { + "api": { + "token": "", + "host": "qase.io" + }, + "batch": { + "size": 100 + }, + "statusFilter": ["passed"], + "showPublicReportLink": true + }, + "testops_multi": { + "default_project": "DEMO1", + "projects": [ + { + "code": "DEMO1", + "run": { + "title": "DEMO1 Multi-Project Run", + "description": "Test run for DEMO1 project", + "complete": true, + "tags": ["staging", "regression"], + "externalLink": { + "type": "jiraCloud", + "link": "PROJ-123" + } + }, + "plan": { + "id": 1 + }, + "environment": "staging" + }, + { + "code": "DEMO2", + "run": { + "title": "DEMO2 Multi-Project Run", + "description": "Test run for DEMO2 project", + "complete": true, + "tags": ["production", "regression"] + }, + "environment": "production" + } + ] + } +} +``` + +### Environment Variables Example + +```bash +# Common settings +export QASE_MODE="testops" +export QASE_FALLBACK="report" +export QASE_ENVIRONMENT="local" +export QASE_DEBUG="true" +export QASE_ROOT_SUITE="MyTestSuite" +export QASE_EXCLUDE_PARAMS="password,token" +export QASE_STATUS_MAPPING="invalid=failed,skipped=passed" + +# Logging configuration +export QASE_LOGGING_CONSOLE="true" +export QASE_LOGGING_FILE="false" + +# Report mode configuration +export QASE_REPORT_DRIVER="local" +export QASE_REPORT_CONNECTION_PATH="./build/qase-report" +export QASE_REPORT_CONNECTION_FORMAT="json" + +# TestOps configuration (single project) +export QASE_TESTOPS_API_TOKEN="" +export QASE_TESTOPS_API_HOST="qase.io" +export QASE_TESTOPS_PROJECT="DEMO" +export QASE_TESTOPS_RUN_TITLE="My Test Run" +export QASE_TESTOPS_RUN_DESCRIPTION="Test run description" +export QASE_TESTOPS_RUN_COMPLETE="true" +export QASE_TESTOPS_RUN_TAGS="tag1,tag2" +export QASE_TESTOPS_RUN_EXTERNAL_LINK='{"type":"jiraCloud","link":"PROJ-123"}' +export QASE_TESTOPS_PLAN_ID="1" +export QASE_TESTOPS_BATCH_SIZE="100" +export QASE_TESTOPS_DEFECT="false" +export QASE_TESTOPS_STATUS_FILTER="passed,skipped" +export QASE_TESTOPS_SHOW_PUBLIC_REPORT_LINK="true" + +# TestOps configurations +export QASE_TESTOPS_CONFIGURATIONS_VALUES='[{"name":"browser","value":"chrome"},{"name":"os","value":"linux"}]' +export QASE_TESTOPS_CONFIGURATIONS_CREATE_IF_NOT_EXISTS="true" + +# Pytest-specific +export QASE_PYTEST_CAPTURE_LOGS="true" +export QASE_PYTEST_XFAIL_STATUS_XFAIL="Skipped" +export QASE_PYTEST_XFAIL_STATUS_XPASS="Passed" + +# Multi-project configuration (default project only) +export QASE_TESTOPS_MULTI_DEFAULT_PROJECT="DEMO1" +``` + +## Multi-Project Support + +The multi-project feature allows you to send test results to multiple Qase projects simultaneously, with different test case IDs for each project. This is useful when: + +* You need to report the same test to different projects +* Different projects track the same functionality with different test case IDs +* You want to maintain separate test runs for different environments or teams + +### How It Works + +1. Configure multiple projects in `testops_multi.projects` array +2. Each project can have its own run configuration (title, description, tags, plan, environment) +3. Use framework-specific annotations to map test cases to projects: + * **Pytest**: Use `@qase.project_id()` decorator + * **Behave**: Use `@qase.project_id.PROJECT_CODE:IDS` tag format + * **Robot Framework**: Use `Q-PROJECT.PROJECT_CODE-IDS` tag format + * **Tavern**: Use `QaseProjectID.PROJECT_CODE=IDS` format in test names +4. Tests without explicit project mapping will be sent to the `default_project` + +### Framework-Specific Documentation + +For detailed framework-specific documentation on multi-project support, see: + +* **[Pytest Multi-Project Guide](../qase-pytest/docs/MULTI_PROJECT.md)** - Detailed guide for using multi-project support with Pytest, including decorator usage, parametrized tests, and test classes +* **[Behave Multi-Project Guide](../qase-behave/docs/MULTI_PROJECT.md)** - Detailed guide for using multi-project support with Behave, including tag formats, feature-level tags, and scenario mapping +* **[Robot Framework Multi-Project Guide](../qase-robotframework/docs/MULTI_PROJECT.md)** - Detailed guide for using multi-project support with Robot Framework, including tag formats, suite-level tags, and parameter handling +* **[Tavern Multi-Project Guide](../qase-tavern/docs/MULTI_PROJECT.md)** - Detailed guide for using multi-project support with Tavern, including test name formats, extraction rules, and troubleshooting + +### Example Usage + +For detailed examples, see the [multi-project examples directory](../examples/multiproject/). + +## Status Mapping + +You can map test result statuses to different values using the `statusMapping` configuration option. This is useful when you want to change how certain statuses are reported to Qase. + +Example: + +```json +{ + "statusMapping": { + "invalid": "failed", + "skipped": "passed" + } +} +``` + +This will map: + +* `invalid` status → `failed` in Qase +* `skipped` status → `passed` in Qase + +## Status Filtering + +You can filter out test results by status using the `testops.statusFilter` configuration option. Results with statuses in the filter list will not be sent to Qase. + +Example: + +```json +{ + "testops": { + "statusFilter": ["passed", "skipped"] + } +} +``` + +This will exclude all `passed` and `skipped` results from being reported to Qase. + +## External Links + +You can associate external links (e.g., Jira tickets) with test runs using the `testops.run.externalLink` configuration. + +Example: + +```json +{ + "testops": { + "run": { + "externalLink": { + "type": "jiraCloud", + "link": "PROJ-123" + } + } + } +} +``` + +Supported types: + +* `jiraCloud` - For Jira Cloud +* `jiraServer` - For Jira Server + +## Configurations + +You can specify test run configurations that will be created or found in Qase TestOps. + +Example: + +```json +{ + "testops": { + "configurations": { + "values": [ + { + "name": "browser", + "value": "chrome" + }, + { + "name": "os", + "value": "linux" + } + ], + "createIfNotExists": true + } + } +} +``` + +If `createIfNotExists` is `true`, configuration groups and values will be created automatically if they don't exist. diff --git a/qase-robotframework/README.md b/qase-robotframework/README.md index 29e8cb8e..91f17ce6 100644 --- a/qase-robotframework/README.md +++ b/qase-robotframework/README.md @@ -22,8 +22,8 @@ Qase Robot Framework Reporter is configured in multiple ways: Environment variables override the values given in the config file. -Configuration options are described in the -[configuration reference](docs/CONFIGURATION.md). +For complete configuration reference, see the [qase-python-commons README](../qase-python-commons/README.md) which contains all available configuration options. + ### Example: qase.config.json @@ -71,7 +71,7 @@ For detailed instructions on using annotations and methods, refer to [Usage](doc Qase Robot Framework Reporter supports sending test results to multiple Qase projects simultaneously. You can specify different test case IDs for each project using the `Q-PROJECT.PROJECT_CODE-IDS` tag format. -For detailed information and examples, see the [Multi-Project Support](docs/usage.md#multi-project-support) section in the usage documentation. +For detailed information, configuration, and examples, see the [Multi-Project Support Guide](docs/MULTI_PROJECT.md). ### Link tests with test cases in Qase TestOps diff --git a/qase-robotframework/docs/CONFIGURATION.md b/qase-robotframework/docs/CONFIGURATION.md deleted file mode 100644 index a45c811d..00000000 --- a/qase-robotframework/docs/CONFIGURATION.md +++ /dev/null @@ -1,104 +0,0 @@ -# Qase Robot Framework Plugin configuration - -Qase Robot Framework Reporter is configured in multiple ways: - -- using a config file `qase.config.json` -- using environment variables - -Environment variables override the values given in the config file. - -## Configuration options - -| Description | Config file | Environment variable | Default value | Required | Possible values | -|-------------------------------------------|----------------------------|---------------------------------|-----------------------------------------|----------|----------------------------| -| **Common** | -| Main reporting mode | `mode` | `QASE_MODE` | `off` | No | `testops`, `report`, `off` | -| Fallback reporting mode | `fallback` | `QASE_FALLBACK` | `report` | No | `testops`, `report`, `off` | -| Execution plan path | `executionPlan.path` | `QASE_EXECUTION_PLAN_PATH` | `./build/qase-execution-plan.json` | No | Any string | -| Qase environment | `environment` | `QASE_ENVIRONMENT` | `local` | No | Any string | -| Root suite | `rootSuite` | `QASE_ROOT_SUITE` | | No | Any string | -| Debug logs | `debug` | `QASE_DEBUG` | `False` | No | `True`, `False` | -| Exclude parameters from test results | `excludeParams` | `QASE_EXCLUDE_PARAMS` | None, don't exclude any parameters | No | Comma-separated list of parameter names | -| **Logging configuration** | -| Enable/disable console logging output | `logging.console` | `QASE_LOGGING_CONSOLE` | `true` | No | `true`, `false` | -| Enable/disable file logging output | `logging.file` | `QASE_LOGGING_FILE` | `false` (true when debug=true) | No | `true`, `false` | -| **Qase TestOps mode configuration** | -| Qase project code | `testops.project` | `QASE_TESTOPS_PROJECT` | | Yes | Any string | -| Qase API token | `testops.api.token` | `QASE_TESTOPS_API_TOKEN` | | Yes | Any string | -| Qase API host. For enterprise users, specify address: `example.qase.io` | `testops.api.host` | `QASE_TESTOPS_API_HOST` | `qase.io` | No | Any string | -| Title of the Qase test run | `testops.run.title` | `QASE_TESTOPS_RUN_TITLE` | `Automated Run {current date and time}` | No | Any string | -| Description of the Qase test run | `testops.run.description` | `QASE_TESTOPS_RUN_DESCRIPTION` | None, leave empty | No | Any string | -| Create test run using a test plan | `testops.plan.id` | `QASE_TESTOPS_PLAN_ID` | None, don't use plans for the test run | No | Any integer | -| Complete test run after running tests | `testops.run.complete` | `QASE_TESTOPS_RUN_COMPLETE` | `True` | No | `true`, `false` | -| ID of the Qase test run to report results | `testops.run.id` | `QASE_TESTOPS_RUN_ID` | None, create a new test run | No | Any integer | -| External link type for test run | `testops.run.externalLink.type` | `QASE_TESTOPS_RUN_EXTERNAL_LINK_TYPE` | None, don't add external link | No | `jiraCloud`, `jiraServer` | -| External link URL for test run | `testops.run.externalLink.link` | `QASE_TESTOPS_RUN_EXTERNAL_LINK_URL` | None, don't add external link | No | Any string (e.g., "PROJ-1234") | -| Batch size for uploading test results | `testops.batch.size` | `QASE_TESTOPS_BATCH_SIZE` | 200 | No | 1 to 2000 | -| Create defects in Qase | `testops.defect` | `QASE_TESTOPS_DEFECT` | `False`, don't create defects | No | `True`, `False` | -| Test run tags | `testops.run.tags` | `QASE_TESTOPS_RUN_TAGS` | None, don't add any tags | No | Comma-separated list of tags | -| Test run configurations | `testops.configurations.values` | `QASE_TESTOPS_CONFIGURATIONS_VALUES` | None, don't add any configurations | No | Format: "group1=value1,group2=value2" | -| Create configurations if not exists | `testops.configurations.createIfNotExists` | `QASE_TESTOPS_CONFIGURATIONS_CREATE_IF_NOT_EXISTS` | `False`, don't create configurations | No | `True`, `False` | -| Filter results by status | `testops.statusFilter` | `QASE_TESTOPS_STATUS_FILTER` | None, don't filter any results | No | Comma-separated list of statuses | -| **Qase Report mode configuration** | -| Local path to store report | `report.connection.path` | `QASE_REPORT_CONNECTION_PATH` | `./build/qase-report` | No | Any string | -| Report format | `report.connection.format` | `QASE_REPORT_CONNECTION_FORMAT` | `json` | No | `json`, `jsonp` | -| Driver used for report mode | `report.driver` | `QASE_REPORT_DRIVER` | `local` | No | `local` | - -## Example `qase.config.json` config: - -```json -{ - "mode": "testops", - "fallback": "report", - "debug": false, - "environment": "local", - "logging": { - "console": true, - "file": false - }, - "report": { - "driver": "local", - "connection": { - "local": { - "path": "./build/qase-report", - "format": "json" - } - } - }, - "testops": { - "api": { - "token": "", - "host": "qase.io" - }, - "run": { - "title": "Regress run", - "description": "Regress run description", - "complete": true, - "tags": ["tag1", "tag2"], - "externalLink": { - "type": "jiraCloud", - "link": "PROJ-1234" - } - }, - "defect": false, - "project": "", - "batch": { - "size": 100 - }, - "statusFilter": ["passed", "failed"], - "configurations": { - "values": [ - { - "name": "group1", - "value": "value1" - }, - { - "name": "group2", - "value": "value2" - } - ], - "createIfNotExists": true - } - } -} -``` diff --git a/qase-robotframework/docs/MULTI_PROJECT.md b/qase-robotframework/docs/MULTI_PROJECT.md new file mode 100644 index 00000000..9873c52b --- /dev/null +++ b/qase-robotframework/docs/MULTI_PROJECT.md @@ -0,0 +1,256 @@ +# Multi-Project Support in Robot Framework + +Qase Robot Framework Reporter supports sending test results to multiple Qase projects simultaneously. This feature allows you to report the same test execution to different projects with different test case IDs, which is useful when: + +* You need to report the same test to different projects +* Different projects track the same functionality with different test case IDs +* You want to maintain separate test runs for different environments or teams + +## Configuration + +For detailed configuration options, refer to the [qase-python-commons README](../../qase-python-commons/README.md#multi-project-support). + +### Basic Multi-Project Configuration + +To enable multi-project support, set the mode to `testops_multi` in your `qase.config.json`: + +```json +{ + "mode": "testops_multi", + "testops": { + "api": { + "token": "", + "host": "qase.io" + }, + "batch": { + "size": 100 + } + }, + "testops_multi": { + "default_project": "PROJ1", + "projects": [ + { + "code": "PROJ1", + "run": { + "title": "PROJ1 Test Run", + "description": "Test run for PROJ1 project", + "complete": true + }, + "environment": "staging" + }, + { + "code": "PROJ2", + "run": { + "title": "PROJ2 Test Run", + "description": "Test run for PROJ2 project", + "complete": true + }, + "environment": "production" + } + ] + } +} +``` + +## Using `Q-PROJECT.PROJECT_CODE-IDS` Tags + +In Robot Framework, you use tags to specify multi-project mappings. The format is `Q-PROJECT.PROJECT_CODE-IDS` where `PROJECT_CODE` is the project code and `IDS` is a comma-separated list of test case IDs. + +### Basic Usage + +```robotframework +*** Test Cases *** +# Single project with single ID +Test with single project ID + [Tags] Q-PROJECT.PROJ1-123 + Step 01 + Step 02 + Passed step + +# Single project with multiple IDs +Test with multiple IDs + [Tags] Q-PROJECT.PROJ1-123,124 + Step 01 + Step 02 + Passed step +``` + +### Multiple Projects + +You can add multiple `Q-PROJECT` tags to send results to multiple projects: + +```robotframework +*** Test Cases *** +# Multiple projects, each with single ID +Test for multiple projects + [Tags] Q-PROJECT.PROJ1-123 Q-PROJECT.PROJ2-456 + Step 01 + Step 02 + Passed step + +# Multiple projects, each with multiple IDs +Complex multi-project test + [Tags] Q-PROJECT.PROJ1-123,124 Q-PROJECT.PROJ2-456,457 + Step 01 + Step 02 + Passed step +``` + +### Combining with Other Tags + +You can combine `Q-PROJECT` tags with other Qase tags: + +```robotframework +*** Test Cases *** +Test with project ID and fields + [Tags] Q-PROJECT.PROJ1-123 qase.fields:{"priority":"high","severity":"critical"} + Step 01 + Step 02 + Passed step +``` + +### Tests Without Project Mapping + +If a test doesn't have any `Q-PROJECT` tags, it will be sent to the `default_project` specified in your configuration: + +```robotframework +*** Test Cases *** +Test without project ID + Step 01 + Step 02 + Passed step +``` + +If `default_project` is not specified, the first project from the `projects` array will be used. + +## Tag Format Details + +The tag format is case-insensitive and follows this pattern: + +* Format: `Q-PROJECT.PROJECT_CODE-ID1,ID2,ID3` +* `PROJECT_CODE`: Must match exactly with the project code in your configuration (case-sensitive) +* `IDS`: Comma-separated list of integers (spaces around commas are allowed) +* Multiple tags can be used for multiple projects + +Examples: + +```robotframework +# Valid formats +[Tags] Q-PROJECT.PROJ1-123 +[Tags] Q-PROJECT.PROJ1-123,124,125 +[Tags] Q-PROJECT.PROJ1-123 Q-PROJECT.PROJ2-456 +[Tags] q-project.proj1-123 # Case-insensitive for Q-PROJECT part + +# Invalid formats (will be ignored) +[Tags] Q-PROJECT.proj1-123 # Wrong case for project code (if config has PROJ1) +[Tags] Q-PROJECT.PROJ1:123 # Wrong separator (should be dash, not colon) +``` + +## Test Suites + +You can apply project ID tags at the suite level using `[Tags]` in the Settings section: + +```robotframework +*** Settings *** +[Tags] Q-PROJECT.PROJ1-100 + +*** Test Cases *** +Test Case 1 + Step 01 + Passed step + +Test Case 2 + Step 01 + Passed step +``` + +All test cases in the suite will inherit the project mapping. You can override it for specific test cases: + +```robotframework +*** Settings *** +[Tags] Q-PROJECT.PROJ1-100 + +*** Test Cases *** +Test Case 1 + Step 01 + Passed step + +Test Case 2 + [Tags] Q-PROJECT.PROJ2-200 + Step 01 + Passed step +``` + +## Working with Parameters + +Multi-project support works with parametrized tests: + +```robotframework +*** Test Cases *** +Parametrized test + [Tags] Q-PROJECT.PROJ1-8 + [Template] Check Value + 1 + 2 + 3 + +Check Value + [Arguments] ${value} + Should Be True ${value} > 0 +``` + +## Important Notes + +1. **Project Codes Must Match**: The project codes used in tags must exactly match the codes specified in your `testops_multi.projects` configuration (case-sensitive). + +2. **Test Case IDs**: Each project can have different test case IDs for the same test. This allows you to maintain separate test case tracking in different projects. + +3. **Test Run Creation**: Each project will have its own test run created (or use an existing run if `run.id` is specified in the project configuration). + +4. **Results Distribution**: Test results are sent to all specified projects simultaneously. If a test fails, the failure will be reported to all projects. + +5. **Default Project**: Tests without explicit project mapping will be sent to the `default_project`. If no `default_project` is specified, the first project in the configuration will be used. + +6. **Mode Requirement**: You must set `mode` to `testops_multi` in your configuration file. Using `testops` mode will not work with `Q-PROJECT` tags. + +7. **Tag Parsing**: The `Q-PROJECT` part is case-insensitive, but the project code itself is case-sensitive and must match your configuration exactly. + +8. **Legacy Tags**: The old `Q-{ID}` format (without project specification) will still work in single-project mode, but will not work in multi-project mode. Use `Q-PROJECT.PROJECT_CODE-IDS` instead. + +## Examples + +See the [multi-project examples](../../../examples/multiproject/robot/) directory for complete working examples. + +## Troubleshooting + +### Test results not appearing in projects + +* Verify that `mode` is set to `testops_multi` in your `qase.config.json` +* Check that project codes in tags match exactly (case-sensitive) with configuration +* Ensure all projects are properly configured in `testops_multi.projects` +* Check debug logs for any errors during test run creation +* Verify tag format is correct (dash separator, proper comma separation) + +### Tests sent to wrong project + +* Verify the `default_project` setting if tests don't have explicit project mapping +* Check that project codes are case-sensitive and match exactly +* Review suite-level tags that might be affecting test case mapping + +### Multiple test runs created + +* This is expected behavior - each project gets its own test run +* To use existing runs, specify `run.id` in each project's configuration + +### Tag not recognized + +* Ensure the tag starts with `Q-PROJECT.` (case-insensitive) +* Check that a dash (`-`) is used to separate project code from IDs (not colon) +* Verify project code matches exactly (case-sensitive) with configuration +* Ensure IDs are comma-separated + +### Old Q-{ID} tags not working + +* In `testops_multi` mode, you must use `Q-PROJECT.PROJECT_CODE-IDS` format +* Old `Q-{ID}` tags will only work in single-project `testops` mode +* Migrate your tags to the new format for multi-project support diff --git a/qase-robotframework/docs/usage.md b/qase-robotframework/docs/usage.md index 37e1a2cd..9e04d4c5 100644 --- a/qase-robotframework/docs/usage.md +++ b/qase-robotframework/docs/usage.md @@ -2,6 +2,8 @@ This guide demonstrates how to integrate Qase with Robot Framework, providing instructions on how to add Qase IDs, fields, and other metadata to your test cases. +> **Configuration:** For complete configuration reference including all available options, environment variables, and examples, see the [qase-python-commons README](../../qase-python-commons/README.md). + ## Adding QaseID to a Test To associate a QaseID with a test in Robot Framework, use the `Q-{ID}` tag format. @@ -23,33 +25,9 @@ Test with multiple QaseIDs ### Multi-Project Support -To send test results to multiple projects with different test case IDs, use the `Q-PROJECT.PROJECT_CODE-IDS` tag format: +Qase Robot Framework Reporter supports sending test results to multiple Qase projects simultaneously with different test case IDs for each project. -```robotframework -*** Test Cases *** -# Single project with multiple IDs -Test with project IDs - [Tags] Q-PROJECT.PROJ1-123,124 - Step 01 - Step 02 - Passed step - -# Multiple projects with different IDs -Test for multiple projects - [Tags] Q-PROJECT.PROJ1-123 Q-PROJECT.PROJ2-456 - Step 01 - Step 02 - Passed step - -# Multiple projects with multiple IDs each -Complex multi-project test - [Tags] Q-PROJECT.PROJ1-123,124 Q-PROJECT.PROJ2-456,457 - Step 01 - Step 02 - Passed step -``` - -**Note:** When using `Q-PROJECT`, the test results will be sent to all specified projects. Make sure to configure `testops_multi` mode in your `qase.config.json` file. +For detailed information, configuration, examples, and troubleshooting, see the [Multi-Project Support Guide](MULTI_PROJECT.md). --- @@ -180,6 +158,8 @@ User Login Test ## Advanced Configuration +For complete configuration options including parallel execution, environment variables, and all other settings, see the [qase-python-commons README](../../qase-python-commons/README.md) and [Robot Framework Configuration Reference](CONFIGURATION.md). + ### Parallel Execution For parallel execution with pabot, the plugin automatically handles worker coordination: diff --git a/qase-tavern/README.md b/qase-tavern/README.md index b93e5c0d..6fd782be 100644 --- a/qase-tavern/README.md +++ b/qase-tavern/README.md @@ -29,7 +29,7 @@ For detailed instructions on using annotations and methods, refer to [Usage](doc Qase Tavern Reporter supports sending test results to multiple Qase projects simultaneously. You can specify different test case IDs for each project using the `QaseProjectID.PROJECT_CODE=IDS` format in test names. -For detailed information and examples, see the [Multi-Project Support](docs/usage.md#multi-project-support) section in the usage documentation. +For detailed information, configuration, and examples, see the [Multi-Project Support Guide](docs/MULTI_PROJECT.md). For example: @@ -66,8 +66,7 @@ Qase Tavern Reporter is configured in multiple ways: Environment variables override the values given in the config file, and command line options override both other values. -Configuration options are described in the -[configuration reference](docs/CONFIGURATION.md). +For complete configuration reference, see the [qase-python-commons README](../qase-python-commons/README.md) which contains all available configuration options. ### Example: qase.config.json diff --git a/qase-tavern/docs/CONFIGURATION.md b/qase-tavern/docs/CONFIGURATION.md deleted file mode 100644 index 554dd34a..00000000 --- a/qase-tavern/docs/CONFIGURATION.md +++ /dev/null @@ -1,107 +0,0 @@ -# Qase Tavern Plugin configuration - -Qase Tavern Reporter is configured in multiple ways: - -- using a config file `qase.config.json` -- using environment variables -- using command line options - -Environment variables override the values given in the config file, -and command line options override both other values. - -## Configuration options - -| Description | Config file | Environment variable | CLI option | Default value | Required | Possible values | -|-------------------------------------------|----------------------------|---------------------------------|-----------------------------------|-----------------------------------------|----------|----------------------------| -| **Common** | -| Main reporting mode | `mode` | `QASE_MODE` | `--qase-mode` | `off` | No | `testops`, `report`, `off` | -| Fallback reporting mode | `fallback` | `QASE_FALLBACK` | `--qase-fallback` | `report` | No | `testops`, `report`, `off` | -| Execution plan path | `executionPlan.path` | `QASE_EXECUTION_PLAN_PATH` | `--qase-execution-plan-path` | `./build/qase-execution-plan.json` | No | Any string | -| Qase environment | `environment` | `QASE_ENVIRONMENT` | `--qase-environment` | `local` | No | Any string | -| Root suite | `rootSuite` | `QASE_ROOT_SUITE` | `--qase-root-suite` | | No | Any string | -| Debug logs | `debug` | `QASE_DEBUG` | `--qase-debug` | false | No | `true`, `false` | -| Exclude parameters from test results | `excludeParams` | `QASE_EXCLUDE_PARAMS` | `--qase-exclude-params` | None | No | Comma-separated list of parameter names | -| **Logging configuration** | -| Enable/disable console logging output | `logging.console` | `QASE_LOGGING_CONSOLE` | `--qase-logging-console` | `true` | No | `true`, `false` | -| Enable/disable file logging output | `logging.file` | `QASE_LOGGING_FILE` | `--qase-logging-file` | `false` (true when debug=true) | No | `true`, `false` | -| **Qase TestOps mode configuration** | -| Qase project code | `testops.project` | `QASE_TESTOPS_PROJECT` | `--qase-testops-project` | | Yes | Any string | -| Qase API token | `testops.api.token` | `QASE_TESTOPS_API_TOKEN` | `--qase-testops-api-token` | | Yes | Any string | -| Qase API host. For enterprise users, specify address: `example.qase.io` | `testops.api.host` | `QASE_TESTOPS_API_HOST` | `--qase-testops-api-host` | `qase.io` | No | Any string | -| Title of the Qase test run | `testops.run.title` | `QASE_TESTOPS_RUN_TITLE` | `--qase-testops-run-title` | `Automated Run {current date and time}` | No | Any string | -| Description of the Qase test run | `testops.run.description` | `QASE_TESTOPS_RUN_DESCRIPTION` | `--qase-testops-run-description` | None, leave empty | No | Any string | -| Create test run using a test plan | `testops.plan.id` | `QASE_TESTOPS_PLAN_ID` | `--qase-testops-plan-id` | None, don't use plans for the test run | No | Any integer | -| Complete test run after running tests | `testops.run.complete` | `QASE_TESTOPS_RUN_COMPLETE` | `--qase-testops-run-complete` | `True` | No | `true`, `false` | -| ID of the Qase test run to report results | `testops.run.id` | `QASE_TESTOPS_RUN_ID` | `--qase-testops-run-id` | None, create a new test run | No | Any integer | -| External link type for test run | `testops.run.externalLink.type` | `QASE_TESTOPS_RUN_EXTERNAL_LINK_TYPE` | `--qase-testops-run-external-link-type` | None, don't add external link | No | `jiraCloud`, `jiraServer` | -| External link URL for test run | `testops.run.externalLink.link` | `QASE_TESTOPS_RUN_EXTERNAL_LINK_URL` | `--qase-testops-run-external-link-url` | None, don't add external link | No | Any string (e.g., "PROJ-1234") | -| Batch size for uploading test results | `testops.batch.size` | `QASE_TESTOPS_BATCH_SIZE` | `--qase-testops-batch-size` | 200 | No | 1 to 2000 | -| Create defects in Qase | `testops.defect` | `QASE_TESTOPS_DEFECT` | `--qase-testops-defect` | `False`, don't create defects | No | `True`, `False` | -| Tags for test run | `testops.run.tags` | `QASE_TESTOPS_RUN_TAGS` | `--qase-testops-run-tags` | None | No | Comma-separated list of tags | -| Test run configurations | `testops.configurations.values` | `QASE_TESTOPS_CONFIGURATIONS_VALUES` | `--qase-testops-configurations-values` | None, don't add any configurations | No | Format: "group1=value1,group2=value2" | -| Create configurations if not exists | `testops.configurations.createIfNotExists` | `QASE_TESTOPS_CONFIGURATIONS_CREATE_IF_NOT_EXISTS` | `--qase-testops-configurations-create-if-not-exists` | `False`, don't create configurations | No | `True`, `False` | -| Filter results by status | `testops.statusFilter` | `QASE_TESTOPS_STATUS_FILTER` | `--qase-testops-status-filter` | None, don't filter any results | No | Comma-separated list of statuses | -| Map test result statuses | `statusMapping` | `STATUS_MAPPING` | `--qase-status-mapping` | None, don't map any statuses | No | Format: 'source1=target1,source2=target2' | -| **Qase Report mode configuration** | -| Local path to store report | `report.connection.path` | `QASE_REPORT_CONNECTION_PATH` | `--qase-report-connection-path` | `./build/qase-report` | No | Any string | -| Report format | `report.connection.format` | `QASE_REPORT_CONNECTION_FORMAT` | `--qase-report-connection-format` | `json` | No | `json`, `jsonp` | -| Driver used for report mode | `report.driver` | `QASE_REPORT_DRIVER` | `--qase-report-driver` | `local` | No | `local` | - -## Example `qase.config.json` config: - -```json -{ - "mode": "testops", - "fallback": "report", - "debug": false, - "environment": "local", - "logging": { - "console": true, - "file": false - }, - "report": { - "driver": "local", - "connection": { - "local": { - "path": "./build/qase-report", - "format": "json" - } - } - }, - "testops": { - "api": { - "token": "", - "host": "qase.io" - }, - "run": { - "title": "Regress run", - "description": "Regress run description", - "complete": true, - "tags": ["tag1", "tag2"], - "externalLink": { - "type": "jiraCloud", - "link": "PROJ-1234" - } - }, - "defect": false, - "project": "", - "batch": { - "size": 100 - }, - "statusFilter": ["passed", "failed"], - "configurations": { - "values": [ - { - "name": "group1", - "value": "value1" - }, - { - "name": "group2", - "value": "value2" - } - ], - "createIfNotExists": true - } - } -} -``` diff --git a/qase-tavern/docs/MULTI_PROJECT.md b/qase-tavern/docs/MULTI_PROJECT.md new file mode 100644 index 00000000..736dda25 --- /dev/null +++ b/qase-tavern/docs/MULTI_PROJECT.md @@ -0,0 +1,271 @@ +# Multi-Project Support in Tavern + +Qase Tavern Reporter supports sending test results to multiple Qase projects simultaneously. This feature allows you to report the same test execution to different projects with different test case IDs, which is useful when: + +* You need to report the same test to different projects +* Different projects track the same functionality with different test case IDs +* You want to maintain separate test runs for different environments or teams + +## Configuration + +For detailed configuration options, refer to the [qase-python-commons README](../../qase-python-commons/README.md#multi-project-support). + +### Basic Multi-Project Configuration + +To enable multi-project support, set the mode to `testops_multi` in your `qase.config.json`: + +```json +{ + "mode": "testops_multi", + "testops": { + "api": { + "token": "", + "host": "qase.io" + }, + "batch": { + "size": 100 + } + }, + "testops_multi": { + "default_project": "PROJ1", + "projects": [ + { + "code": "PROJ1", + "run": { + "title": "PROJ1 Test Run", + "description": "Test run for PROJ1 project", + "complete": true + }, + "environment": "staging" + }, + { + "code": "PROJ2", + "run": { + "title": "PROJ2 Test Run", + "description": "Test run for PROJ2 project", + "complete": true + }, + "environment": "production" + } + ] + } +} +``` + +## Using `QaseProjectID.PROJECT_CODE=IDS` in Test Names + +In Tavern, you specify multi-project mappings directly in the test name using the format `QaseProjectID.PROJECT_CODE=IDS` where `PROJECT_CODE` is the project code and `IDS` is a comma-separated list of test case IDs. + +### Basic Usage + +```yaml +--- +# Single project with single ID +test_name: QaseProjectID.PROJ1=123 Get user by ID + +stages: + - name: Get user + request: + url: https://api.example.com/users/1 + method: GET + response: + status_code: 200 + +--- +# Single project with multiple IDs +test_name: QaseProjectID.PROJ1=123,124 Get user by ID + +stages: + - name: Get user + request: + url: https://api.example.com/users/1 + method: GET + response: + status_code: 200 +``` + +### Multiple Projects + +You can specify multiple `QaseProjectID` entries in the test name to send results to multiple projects: + +```yaml +--- +# Multiple projects, each with single ID +test_name: QaseProjectID.PROJ1=123 QaseProjectID.PROJ2=456 Get user by ID + +stages: + - name: Get user + request: + url: https://api.example.com/users/1 + method: GET + response: + status_code: 200 + +--- +# Multiple projects, each with multiple IDs +test_name: QaseProjectID.PROJ1=123,124 QaseProjectID.PROJ2=456,457 Get user by ID + +stages: + - name: Get user + request: + url: https://api.example.com/users/1 + method: GET + response: + status_code: 200 +``` + +### Combining with QaseID + +You cannot use both `QaseID` and `QaseProjectID` in the same test name. When using multi-project mode, always use `QaseProjectID`: + +```yaml +--- +# Correct: Using QaseProjectID for multi-project +test_name: QaseProjectID.PROJ1=123 QaseProjectID.PROJ2=456 Get user by ID + +stages: + - name: Get user + request: + url: https://api.example.com/users/1 + method: GET + response: + status_code: 200 + +--- +# Incorrect: Mixing QaseID and QaseProjectID (QaseID will be ignored) +test_name: QaseID=123 QaseProjectID.PROJ2=456 Get user by ID + +stages: + - name: Get user + request: + url: https://api.example.com/users/1 + method: GET + response: + status_code: 200 +``` + +### Tests Without Project Mapping + +If a test doesn't have any `QaseProjectID` in its name, it will be sent to the `default_project` specified in your configuration: + +```yaml +--- +test_name: Get user by ID + +stages: + - name: Get user + request: + url: https://api.example.com/users/1 + method: GET + response: + status_code: 200 +``` + +If `default_project` is not specified, the first project from the `projects` array will be used. + +## Test Name Format Details + +The format is case-insensitive and follows this pattern: + +* Format: `QaseProjectID.PROJECT_CODE=ID1,ID2,ID3` +* `PROJECT_CODE`: Must match exactly with the project code in your configuration (case-sensitive) +* `IDS`: Comma-separated list of integers (spaces around commas are allowed) +* Multiple `QaseProjectID` entries can be used for multiple projects +* The rest of the test name after `QaseProjectID` entries will be used as the test title + +Examples: + +```yaml +# Valid formats +test_name: QaseProjectID.PROJ1=123 Get user by ID +test_name: QaseProjectID.PROJ1=123,124,125 Get user by ID +test_name: QaseProjectID.PROJ1=123 QaseProjectID.PROJ2=456 Get user by ID +test_name: qaseprojectid.proj1=123 Get user by ID # Case-insensitive + +# Invalid formats (will be ignored or cause errors) +test_name: QaseProjectID.proj1=123 Get user by ID # Wrong case for project code (if config has PROJ1) +test_name: QaseProjectID.PROJ1:123 Get user by ID # Wrong separator (should be =, not :) +``` + +## Order of Extraction + +The reporter extracts project IDs from the test name in the following order: + +1. First, all `QaseProjectID.PROJECT_CODE=IDS` patterns are extracted +2. If no project IDs are found, `QaseID=IDS` is extracted (for single-project mode) +3. The remaining text after extraction is used as the test title + +Example: + +```yaml +--- +test_name: QaseProjectID.PROJ1=123 QaseProjectID.PROJ2=456 Get user by ID +``` + +This will: +* Extract project mapping: `PROJ1` → `[123]`, `PROJ2` → `[456]` +* Use "Get user by ID" as the test title + +## Important Notes + +1. **Project Codes Must Match**: The project codes used in test names must exactly match the codes specified in your `testops_multi.projects` configuration (case-sensitive). + +2. **Test Case IDs**: Each project can have different test case IDs for the same test. This allows you to maintain separate test case tracking in different projects. + +3. **Test Run Creation**: Each project will have its own test run created (or use an existing run if `run.id` is specified in the project configuration). + +4. **Results Distribution**: Test results are sent to all specified projects simultaneously. If a test fails, the failure will be reported to all projects. + +5. **Default Project**: Tests without explicit project mapping will be sent to the `default_project`. If no `default_project` is specified, the first project in the configuration will be used. + +6. **Mode Requirement**: You must set `mode` to `testops_multi` in your configuration file. Using `testops` mode will not work with `QaseProjectID` format. + +7. **Name Parsing**: The `QaseProjectID` part is case-insensitive, but the project code itself is case-sensitive and must match your configuration exactly. + +8. **Spaces**: Spaces around the `=` sign and commas are allowed and will be trimmed automatically. + +9. **Legacy Format**: The old `QaseID=IDS` format will still work in single-project mode, but will not work in multi-project mode. Use `QaseProjectID.PROJECT_CODE=IDS` instead. + +## Examples + +See the [multi-project examples](../../../examples/multiproject/tavern/) directory for complete working examples. + +## Troubleshooting + +### Test results not appearing in projects + +* Verify that `mode` is set to `testops_multi` in your `qase.config.json` +* Check that project codes in test names match exactly (case-sensitive) with configuration +* Ensure all projects are properly configured in `testops_multi.projects` +* Check debug logs for any errors during test run creation +* Verify test name format is correct (equals sign separator, proper comma separation) + +### Tests sent to wrong project + +* Verify the `default_project` setting if tests don't have explicit project mapping +* Check that project codes are case-sensitive and match exactly +* Review test names to ensure `QaseProjectID` format is correct + +### Multiple test runs created + +* This is expected behavior - each project gets its own test run +* To use existing runs, specify `run.id` in each project's configuration + +### Project ID not recognized + +* Ensure the format is `QaseProjectID.PROJECT_CODE=IDS` (case-insensitive for `QaseProjectID` part) +* Check that an equals sign (`=`) is used to separate project code from IDs (not colon) +* Verify project code matches exactly (case-sensitive) with configuration +* Ensure IDs are comma-separated + +### Old QaseID format not working + +* In `testops_multi` mode, you must use `QaseProjectID.PROJECT_CODE=IDS` format +* Old `QaseID=IDS` format will only work in single-project `testops` mode +* Migrate your test names to the new format for multi-project support + +### Test title is empty or incorrect + +* The test title is extracted from the remaining text after removing all `QaseProjectID` patterns +* Ensure there is descriptive text after the project ID specifications +* Example: `QaseProjectID.PROJ1=123 Get user by ID` → title will be "Get user by ID" diff --git a/qase-tavern/docs/usage.md b/qase-tavern/docs/usage.md index ab80c937..f7a2da7c 100644 --- a/qase-tavern/docs/usage.md +++ b/qase-tavern/docs/usage.md @@ -2,6 +2,8 @@ This guide demonstrates how to integrate Qase with Tavern, providing instructions on how to add Qase IDs and other metadata to your API test cases. +> **Configuration:** For complete configuration reference including all available options, environment variables, and examples, see the [qase-python-commons README](../../qase-python-commons/README.md). + --- ## Adding QaseID to a Test @@ -48,47 +50,9 @@ stages: ### Multi-Project Support -To send test results to multiple projects with different test case IDs, use the `QaseProjectID.PROJECT_CODE=IDS` format in the test name: - -```yaml ---- -# Single project with multiple IDs -test_name: QaseProjectID.PROJ1=123,124 Get user by ID - -stages: - - name: Get user - request: - url: https://api.example.com/users/1 - method: GET - response: - status_code: 200 - ---- -# Multiple projects with different IDs -test_name: QaseProjectID.PROJ1=123 QaseProjectID.PROJ2=456 Get user by ID - -stages: - - name: Get user - request: - url: https://api.example.com/users/1 - method: GET - response: - status_code: 200 - ---- -# Multiple projects with multiple IDs each -test_name: QaseProjectID.PROJ1=123,124 QaseProjectID.PROJ2=456,457 Get user by ID - -stages: - - name: Get user - request: - url: https://api.example.com/users/1 - method: GET - response: - status_code: 200 -``` +Qase Tavern Reporter supports sending test results to multiple Qase projects simultaneously with different test case IDs for each project. -**Note:** When using `QaseProjectID`, the test results will be sent to all specified projects. Make sure to configure `testops_multi` mode in your `qase.config.json` file. +For detailed information, configuration, examples, and troubleshooting, see the [Multi-Project Support Guide](MULTI_PROJECT.md). ---