Skip to content

coverage.md

Latest commit

 

History

History
150 lines (115 loc) · 7.08 KB

coverage.md

File metadata and controls

150 lines (115 loc) · 7.08 KB

Coverage Report

After running tests, the PHPUnit extension prints a coverage report. The output format is controlled by the console_output parameter (or OPENAPI_CONSOLE_OUTPUT environment variable).

Coverage is tracked at (method, path, statusCode, contentType) granularity: a GET /v1/pets test that only exercises 200 application/json does not count 404 or application/problem+json as covered. Per-endpoint markers reflect the resolved state across all declared response definitions:

Marker Meaning
✓ / :white_check_mark: All declared (status, content-type) pairs validated
◐ / :large_orange_diamond: Some pairs validated, others uncovered
⚠ / :warning: Pair was skipped (e.g. 5XX matched the default skip pattern)
✗ / :x: No pair validated for this endpoint
· / :information_source: Endpoint reached via request-validation but no response asserted

The report also breaks the coverage rate into two numbers — the strict endpoint rate (all declared responses validated) and the response-level rate (responseCovered / responseTotal).

Console modes

default mode (default)

Shows endpoint summary lines only:

OpenAPI Contract Test Coverage
==================================================

[front] endpoints: 12/45 fully covered (26.7%), 8 partial, 25 uncovered
        responses: 38/120 covered (31.7%), 4 skipped, 78 uncovered
--------------------------------------------------
Legend: ✓=validated  ⚠=skipped  ✗=uncovered  ◐=partial  ·=request-only  *=any/no content-type
  ✓ GET /v1/pets  (3/3 responses)
  ◐ POST /v1/pets  (1/2 responses)
  ◐ DELETE /v1/pets/{petId}  (1/2 responses, 1 skipped)
  ✗ PUT /v1/pets/{petId}  (0/2 responses)

Endpoint markers come from a fixed set: ✓ all-covered, ◐ partial (any combination of validated, skipped, uncovered short of full coverage), · request-only, ✗ uncovered. The ⚠ marker is reserved for per-response sub-rows (skipped responses), never for endpoint summary lines.

all mode

Shows endpoint summaries with per-response sub-rows. Sub-row whitespace is illustrative — the renderer pads statusKey to 5 chars and contentTypeKey to 32 chars:

[front] endpoints: 12/45 fully covered (26.7%), 8 partial, 25 uncovered
        responses: 38/120 covered (31.7%), 4 skipped, 78 uncovered
--------------------------------------------------
Legend: ✓=validated  ⚠=skipped  ✗=uncovered  ◐=partial  ·=request-only  *=any/no content-type
  ✓ GET /v1/pets  (3/3 responses)
      ✓ 200    application/json                  [12]
      ✓ 400    application/problem+json          [1]
      ✓ 422    Application/Problem+JSON          [1]
  ◐ POST /v1/pets  (1/2 responses)
      ✓ 201    application/json                  [3]
      ✗ 422    application/problem+json          uncovered
  ◐ DELETE /v1/pets/{petId}  (1/2 responses, 1 skipped)
      ✓ 204    *                                 [2]
      ⚠ 5XX    *                                 skipped: status 503 matched skip pattern 5\d\d

uncovered_only mode

Shows sub-rows only for partial / uncovered endpoints, keeping fully-covered ones compact:

[front] endpoints: 12/45 fully covered (26.7%), 8 partial, 25 uncovered
        responses: 38/120 covered (31.7%), 4 skipped, 78 uncovered
--------------------------------------------------
Legend: ✓=validated  ⚠=skipped  ✗=uncovered  ◐=partial  ·=request-only  *=any/no content-type
  ✓ GET /v1/pets  (3/3 responses)
  ◐ POST /v1/pets  (1/2 responses)
      ✗ 422    application/problem+json          uncovered
  ✗ PUT /v1/pets/{petId}  (0/2 responses)
      ✗ 200    application/json                  uncovered
      ✗ 404    application/problem+json          uncovered

active_only mode

Useful for the local TDD loop with a multi-spec setup (e.g. specs="front,store,admin"). Specs that no test in this run touched are collapsed to a single line, so a focused single-test run no longer has to scroll past hundreds of ✗ uncovered rows for unrelated specs. Specs with at least one validated, skipped, or request-only observation render the same one-line-per-endpoint view as default:

[front] no test activity (373 endpoints, 894 responses in spec)
[store] no test activity (148 endpoints, 312 responses in spec)

[admin] endpoints: 1/72 fully covered (1.4%), 0 partial, 71 uncovered
        responses: 1/172 covered (0.6%), 0 skipped, 171 uncovered
--------------------------------------------------
Legend: ✓=validated  ⚠=skipped  ✗=uncovered  ◐=partial  ·=request-only  *=any/no content-type
  ✓ GET /v2/admin/early_accesses  (1/1 responses)
  ✗ POST /v2/admin/early_accesses  (0/2 responses)
  ...

You can set the mode via phpunit.xml:

<parameter name="console_output" value="uncovered_only"/>

Or via environment variable (takes priority over phpunit.xml):

OPENAPI_CONSOLE_OUTPUT=uncovered_only vendor/bin/phpunit

Coverage threshold gate

Optional CI gate that fails the run when contract coverage drops below a configured percentage — the contract-testing analogue of PHPUnit's own --coverage-threshold. Both metrics are aggregated across every spec listed in specs=:

  • min_endpoint_coverage — percentage of endpoints with all declared (status, content-type) pairs validated.
  • min_response_coverage — percentage of (method, path, status, content-type) rows validated (the same rate the report calls "responses covered").

Default is warn-only: a miss prints [OpenAPI Coverage] WARN: … to stderr but the run exits 0. Flip min_coverage_strict=true to make a miss fail-fast with exit 1.

<extensions>
    <bootstrap class="Studio\OpenApiContractTesting\PHPUnit\OpenApiCoverageExtension">
        <parameter name="spec_base_path" value="openapi/bundled"/>
        <parameter name="specs" value="front,admin"/>
        <parameter name="min_endpoint_coverage" value="80"/>   <!-- percent, optional -->
        <parameter name="min_response_coverage" value="60"/>   <!-- percent, optional -->
        <parameter name="min_coverage_strict" value="true"/>   <!-- default false → warn-only -->
    </bootstrap>
</extensions>

Failure looks like:

[OpenAPI Coverage] FAIL: endpoint coverage 67.4% < threshold 80%.
                         response coverage 71.2% (>= 60%, ok).

Out-of-range or non-numeric values produce a WARNING to stderr and skip that gate (rather than silently treating the misconfiguration as 0%).

For paratest / pest --parallel, the merge CLI accepts the same options as flags:

vendor/bin/openapi-coverage-merge \
    --spec-base-path=openapi/bundled \
    --specs=front,admin \
    --min-endpoint-coverage=80 \
    --min-response-coverage=60 \
    --min-coverage-strict