ci(backend): parallelize full-suite coverage

[Abiorh001]

Intent

Complete WS-CI-001-01 by running the complete backend test inventory across four isolated jobs, then authenticate and combine every result before the existing required coverage gates pass.

Design

• Preflight discovers every canonical module and pytest node and creates a deterministic four-shard manifest.
• Each shard validates the checked-out tree, owns its PostgreSQL database and MinIO instance, executes exact manifest node IDs, and records nodes only after their runtime lifecycle finishes.
• Fixed bundles bind tree SHA, manifest digest, completed nodes, coverage bytes, shard identity, and duration metadata.
• Final Backend / test rejects any upstream or artifact inconsistency, combines coverage, and enforces the unchanged 78 percent global and twelve 90 percent subsystem floors.
• API contract E2E runs concurrently in its own isolated database.

Evidence

• 294 focused planner, agent-gate, and coverage-contract tests pass.
• 188 isolated-runner and coverage-contract tests pass against local PostgreSQL.
• Clean-environment dry run discovers 31 modules and 1,774 nodes at shard weights 443/444/443/444.
• Ruff, compilation, merge-intent, loop-state, stale scans, Markdown links, internal-review evidence, and diff integrity pass.
• All nine internal reviewer tracks pass after valid findings were repaired.

Human review focus

Review exact runtime-node proof, database and MinIO isolation, fixed artifact provenance, cancellation propagation, coverage combination, unchanged thresholds, stable Backend / test identity, and hosted latency versus runner cost.

Trust bundle

.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/reviews/WS-CI-001-01-pr-trust-bundle.md

Merge intent: .agent-loop/merge-intents/WS-CI-001-01.json

Human approval is required before merge. WS-CI-001-02 and WS-ENG-001-04B remain inactive.

Summary by CodeRabbit

• New Features

  • Backend CI now runs the full test suite in deterministic, isolated parallel shards.
  • Added authenticated coverage aggregation and API contract testing within the CI pipeline.
  • Added fail-closed validation for test completeness, coverage integrity, and artifact provenance.
  • Pinned database and service images for more reproducible builds.

• Documentation

  • Added operational guidance for parallel CI runs, evidence bundles, failures, reruns, and rollback procedures.
  • Added initiative plans, risk assessments, decisions, and review evidence.

• Tests

  • Added comprehensive validation for sharding, coverage fan-in, artifact integrity, and workflow configuration.

[coderabbitai]

📝 Walkthrough

Walkthrough

The PR defines WS-CI-001 and replaces the sequential backend test workflow with deterministic four-way sharding, isolated databases, authenticated coverage bundles, concurrent API E2E, and fail-closed coverage fan-in. It adds the shard planner, validation tests, workflow assertions, operational documentation, and review evidence.

Changes

Backend CI acceleration

Layer / File(s)	Summary
Contracts, governance, and evidence .agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/..., .agent-loop/merge-intents/...	Adds initiative discovery, intent, plan, risks, decisions, chunk contracts, status, merge metadata, and review evidence for deterministic full-suite acceleration and deferred routing changes.
Manifest discovery and deterministic planning backend/scripts/ci_test_shards.py, backend/tests/test_ci_test_shards.py	Discovers canonical test modules, collects pytest nodes, normalizes parameterized IDs, builds deterministic weighted manifests, validates digests, and tests invalid inventories and tampering.
Shard execution and authenticated fan-in backend/scripts/ci_test_shards.py, backend/tests/test_ci_test_shards.py	Runs isolated whole-module shards, records collection/completion evidence, writes coverage metadata, validates bundle provenance and filesystem contents, and rejects mismatched or altered evidence.
GitHub Actions orchestration and operations .github/workflows/backend.yml, scripts/test_agent_gates.py, docs/operations_backend_testing.md	Adds preflight, four shards, concurrent API E2E, fail-closed fan-in, pinned service images, coverage gates, workflow regression assertions, and operational guidance.

Estimated code review effort: 5 (Critical) | ~120 minutes

Sequence Diagram(s)

sequenceDiagram
  participant GitHubActions
  participant Preflight
  participant Shards
  participant APIE2E
  participant BackendTest
  GitHubActions->>Preflight: validate tree and create manifest
  Preflight->>Shards: upload immutable manifest
  Preflight->>APIE2E: provide tree identity
  Shards->>Shards: run four isolated test jobs
  APIE2E->>BackendTest: report contract E2E result
  Shards->>BackendTest: upload authenticated coverage bundles
  BackendTest->>BackendTest: validate evidence and combine coverage

Loading

Possibly related PRs

• Flow-Research/workstream#103: Shares the isolated Postgres test-runner foundation used by the sharded workflow.
• Flow-Research/workstream#138: Overlaps in backend workflow execution, coverage reporting, and API E2E wiring.
• Flow-Research/workstream#151: Shares MinIO pinning and S3-related coverage integration points.

Suggested reviewers: abiorh-claw

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (2 warnings)

Check name	Status	Explanation	Resolution
Description check	⚠️ Warning	The description covers intent, design, evidence, and review focus, but omits most required template sections.	Add the missing template sections: Chunk/Goal, Human-Approved Intent, What Changed/Why/Alternatives, Scope Control, Commands Run, Result Summary, Acceptance Criteria, Test Delta, reviewer tables, and CI/gate integrity.
Docstring Coverage	⚠️ Warning	Docstring coverage is 20.75% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (3 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly summarizes the main change: parallelizing backend full-suite coverage.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch codex/ws-eng-002-backend-ci-acceleration

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@backend/scripts/ci_test_shards.py`:
- Around line 388-389: Reduce the --timeout-seconds value in
backend/scripts/ci_test_shards.py at lines 388-389 to below the 90-minute shards
job limit, and reduce the corresponding --timeout-seconds value in
.github/workflows/backend.yml at lines 228-231 to below the 30-minute api_e2e
job limit.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 35a7adda-2d02-4291-8090-c352a8c80506

📥 Commits

Reviewing files that changed from the base of the PR and between c12ba1c and 1fa072b.

📒 Files selected for processing (18)

• .agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/CHUNK_MAP.md
• .agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/DECISIONS.md
• .agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/DISCOVERY.md
• .agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/INTENT.md
• .agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/PLAN.md
• .agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/RISKS.md
• .agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/STATUS.md
• .agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/chunks/WS-CI-001-01-parallel-full-suite-coverage.md
• .agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/chunks/WS-CI-001-02-safe-routing-cache-timing.md
• .agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/reviews/WS-CI-001-01-internal-review-evidence.md
• .agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/reviews/WS-CI-001-01-pr-trust-bundle.md
• .agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/reviews/WS-CI-001-01-preimplementation-plan-review.md
• .agent-loop/merge-intents/WS-CI-001-01.json
• .github/workflows/backend.yml
• backend/scripts/ci_test_shards.py
• backend/tests/test_ci_test_shards.py
• docs/operations_backend_testing.md
• scripts/test_agent_gates.py