Skip to content

ci: shorten PR feedback without reducing test coverage #935

Description

@Brad-Edwards

Problem

The canonical PR verification graph runs roughly 5,100 Python tests and currently takes about 12 minutes. That test volume is appropriate for the repository, but the pipeline makes small changes wait for the complete suite before Sonar starts. Failures that Sonar can identify independently therefore arrive late, and each corrective merge restarts the full feedback cycle.

Observed on PR #932:

  • Full pytest execution: approximately 5,100 tests
  • Verify duration: approximately 12 minutes
  • Sonar quality-gate evaluation starts only after verify completes
  • Small, isolated fixes repeatedly pay the full verification latency

Goal

Reduce time to actionable PR feedback while preserving the full test and quality-gate coverage required before protected-branch integration.

Proposed work

  • Split the Python test suite into deterministic CI shards that run concurrently.
  • Evaluate conservative change-based test selection for an early feedback lane, while retaining the complete suite as a required merge gate.
  • Decouple Sonar analysis from unrelated sequential verification stages; start it as soon as its required source and coverage inputs are available.
  • If coverage is produced by test shards, combine shard coverage artifacts before the final Sonar quality-gate step.
  • Preserve repo-policy, contract, fuzz, integration, supply-chain, and full-suite checks as required gates.
  • Measure current and resulting median and p95 times to first failure and final required-check completion.

Acceptance criteria

  • The full existing test suite remains required before protected-branch integration.
  • Test sharding is deterministic and demonstrably executes every test exactly once per full run.
  • Sonar no longer waits on verification stages unrelated to its required inputs.
  • A small Python-only change receives actionable test/static-analysis feedback materially earlier than the current approximately 12-minute critical path.
  • Coverage reports remain complete and compatible with the Sonar quality gate.
  • CI documentation describes the fast-feedback path, full gate, shard ownership, and failure-reproduction commands.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions