Skip to content

CI: aggregator job as single required check; drop trigger-level paths filter#214

Draft
glwagner wants to merge 2 commits into
mainfrom
glw/ci-aggregator-required-check
Draft

CI: aggregator job as single required check; drop trigger-level paths filter#214
glwagner wants to merge 2 commits into
mainfrom
glw/ci-aggregator-required-check

Conversation

@glwagner
Copy link
Copy Markdown
Member

@glwagner glwagner commented May 6, 2026

What

Restructure CI and Documentation so a single aggregator job (test-summary / docs-summary) is the only required status check, and the workflow always runs (so the aggregator always reports).

Why

Both workflows had trigger-level paths: filters. When a PR touched no in-path files, the whole workflow was skipped. GitHub branch protection treats "never reported" as "still pending" — not as "passed" — so the merge button stayed disabled for non-admins even though the conversation view said "all checks passed". PR #209 hit this; only an admin force-merge got it through.

A required status check that never reports is not the same as one that passed. Skipped/not-triggered jobs do not satisfy the requirement.

How

  • Drop trigger-level paths: from both workflows so the workflow always starts.
  • Add a changes job using dorny/paths-filter@v3 that exposes a boolean output for the same path set the trigger filter used to gate.
  • Gate each previously-conditional job with if: github.event_name != 'pull_request' || needs.changes.outputs.<x> == 'true', so out-of-path PRs skip the heavy jobs (preserving prior behavior) but the workflow itself still runs.
  • Add an if: always() aggregator (test-summary in ci.yml, docs-summary in docs.yml) that passes iff every needed job is success or skipped. Real failures still block the merge; legitimate skips don't.

The matrix in cpu_tests reports skipped only if the entire matrix was skipped — that's the desired behavior, no special handling needed.

⚠️ Branch protection follow-up (UI, not code)

This part has to be done in the GitHub UI — the workflow change alone is not enough. In Settings → Rules → Rulesets (or legacy Branch protection rules for main):

Remove these from "Require status checks to pass":

  • (CPU) - Julia 1.12.5
  • (GPU)
  • Data Downloading - … (each matrix variant)
  • Reactant extension - …
  • Build documentation
  • deploy-docs
  • the old CI aggregator (if listed)
  • any per-job names from the docs workflow

Add only:

  • test-summary
  • docs-summary

Leaving any of the old job names in the required-checks list will reproduce the bug regardless of these workflow changes — that's the load-bearing step.

Out of scope

  • Changing what tests run when. The conditional logic is preserved; only how completion is reported has changed.
  • Migrating from branch protection rules to rulesets (orthogonal — same fix applies to both).

🤖 Generated with Claude Code

Background
----------
Both `CI` and `Documentation` had trigger-level `paths:` filters. When a
PR touched no in-path files, the entire workflow was skipped — which, for
GitHub branch protection, is *not* the same as passing. Required checks
that never report stay pending forever, the PR shows "all checks passed"
in the conversation view, but the merge button is disabled for
non-admins. PR #209 hit this.

Fix
---
- Drop trigger-level `paths:` from both workflows so the workflow always
  starts and the aggregator always reports a status.
- Add a `changes` job using `dorny/paths-filter@v3` that exposes a
  boolean output for the same path set the trigger filter used to gate.
- Gate each previously-conditional job with
  `if: github.event_name \!= 'pull_request' || needs.changes.outputs.<x> == 'true'`,
  so out-of-path PRs skip the heavy jobs (preserving prior behavior) but
  the workflow itself still runs.
- Add a `test-summary` aggregator (and `docs-summary` in docs.yml) that
  runs with `if: always()` and passes iff every needed job is `success`
  or `skipped`. Real failures still block; legitimate skips don't.

Branch protection follow-up (UI, not code)
------------------------------------------
In repo Settings → Rules → Rulesets (or legacy Branch protection rules
for `main`), the required-status-checks list needs to be updated to
match this restructure:

- Remove the per-job names (e.g. `(CPU) - Julia 1.12.5`, `(GPU)`,
  `Data Downloading - …`, `Reactant extension - …`, `Build documentation`,
  `deploy-docs`, plus the old `CI` aggregator).
- Add only:
    - `test-summary`
    - `docs-summary`

Leaving any of the old job names in the required list would reproduce
the bug regardless of these workflow changes.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@codecov
Copy link
Copy Markdown

codecov Bot commented May 6, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@simone-silvestri
Copy link
Copy Markdown
Member

Cool, so in theory the CI does not run here if we don't change the src but there is also no "pending" final check?

@glwagner glwagner marked this pull request as draft May 6, 2026 14:40
@glwagner
Copy link
Copy Markdown
Member Author

glwagner commented May 6, 2026

@giordano does this make sense or is there a better way?

Copy link
Copy Markdown
Member

@giordano giordano left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@giordano does this make sense or is there a better way?

Yeah, I've seen this approach before, I just find it annoying that GitHub marks as pending jobs which are skipped, and I wish they fixed that instead of forcing everybody to work around it.

Comment thread .github/workflows/ci.yml
Comment on lines -328 to +366
- run: echo "All CI jobs passed successfully."
- name: Check that all needed jobs passed or were skipped
run: |
echo '${{ toJSON(needs) }}'
jq -e 'to_entries | all(.value.result == "success" or .value.result == "skipped")' <<< '${{ toJSON(needs) }}'
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's this for?

@navidcy navidcy added the testing 🧪 trust but verify label May 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

testing 🧪 trust but verify

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants