ci: add organization PR validation baseline - #8
Conversation
|
Warning Review limit reached
Next review available in: 51 minutes You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughAdded a reusable, read-only GitHub Actions workflow. It validates workflow files, audits permissions and security patterns, verifies pinned tool checksums, and scans pull-request or full commit history for secrets. ChangesOrganization PR baseline
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant GitHubActions
participant Runner
participant WorkflowAudit
participant Gitleaks
GitHubActions->>Runner: Start validation job
Runner->>WorkflowAudit: Lint and audit workflow files
WorkflowAudit-->>Runner: Return blocking and advisory findings
Runner->>Gitleaks: Scan commit range or full history
Gitleaks-->>Runner: Return secret findings
Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
🧹 Nitpick comments (3)
.github/workflows/organization-pr-baseline.yml (3)
29-35: 🚀 Performance & Scalability | 🔵 TrivialConsider the interaction between
timeout-minutes: 10and full-history scans.Two settings combine here.
fetch-depth: 0clones all history. The gitleaks step falls back to a full-history scan wheneverBASEis empty, which happens for everyworkflow_callinvocation outside a pull-request event.For a large monorepo, the clone plus the full scan can exceed 10 minutes. The job then fails on timeout, and repository owners see a red gate that has nothing to do with a leak. Measure the runtime on the largest repository in the trial cohort before you widen the rollout.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.github/workflows/organization-pr-baseline.yml around lines 29 - 35, Validate the combined runtime of the full-history checkout and gitleaks fallback scan in the organization-pr-baseline workflow, especially for workflow_call invocations with an empty BASE. Measure this on the largest trial repository and adjust timeout-minutes or rollout scope based on the observed runtime so legitimate scans do not fail due to the job timeout.
95-106: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winThis scan logic duplicates
.github/workflows/secret-scan-reusable.yml.The range selection, the zero-SHA guard, the
git rev-parse --verifyfallback, and thegitleaks gitflags match lines 10-52 of.github/workflows/secret-scan-reusable.yml. The two copies will drift.The new copy is the hardened one. It pins the download by checksum and disables credential persistence. The older file uses
actions/checkout@v7and an unverified download. After the staged rollout completes, retire the older reusable workflow or make it call this one.The logic itself is correct. The fallback to a full-history scan when
BASEis empty is the safe direction forworkflow_callinvocations that carry no pull-request context.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.github/workflows/organization-pr-baseline.yml around lines 95 - 106, The secret-scan logic in the pull-request workflow duplicates secret-scan-reusable.yml and must have a single maintained implementation. After the staged rollout, retire the older reusable workflow, or update it to call the hardened scan implementation while preserving its range selection, zero-SHA guard, commit verification fallback, and gitleaks flags.
69-93: 🎯 Functional Correctness | 🔵 Trivial | 🏗️ Heavy liftThe placeholder gate is both over-broad and easy to bypass.
Three concrete gaps exist in the current matcher:
- Scope. Both regexes run over the whole file. A workflow that contains a
CI Summaryjob and, in an unrelated job, the stringecho "CI checks passed"fails the gate. The two matches never need to belong to the same job.- Phrase coupling. The gate only rejects the literal phrase
CI checks passed. A placeholder such asecho ok,true, orexit 0passes.- Position coupling. The regex requires
echoto be the first token afterrun:. A block scalar such asrun: |followed byset -eand thenecho "CI checks passed"is not matched.Parse the YAML and inspect the steps of the
CI Summaryjob instead. That removes all three gaps and makes the rule explainable to repository owners.♻️ Proposed job-scoped implementation
- name: Reject placeholder CI Summary gates shell: python run: | from pathlib import Path - import re import sys + import yaml - placeholder = re.compile( - r"run:\s*(?:[>|][-+]?\s*\n\s*)?echo\s+[\"']?CI checks passed", - re.IGNORECASE, - ) + # A gate is a placeholder when every step only echoes, exits 0, or is a no-op. + def is_placeholder(step): + run = (step.get("run") or "").strip() + if not run: + return False + lines = [ + line.strip() + for line in run.splitlines() + if line.strip() and not line.strip().startswith("#") + ] + trivial = ("echo", "true", "exit 0", "set ", ":") + return all(line.startswith(trivial) for line in lines) + failures = [] for pattern in ("*.yml", "*.yaml"): for path in Path(".github/workflows").glob(pattern): - text = path.read_text(encoding="utf-8") - if re.search(r"name:\s*CI Summary\b", text) and placeholder.search(text): - failures.append(path) + try: + doc = yaml.safe_load(path.read_text(encoding="utf-8")) or {} + except (yaml.YAMLError, UnicodeDecodeError) as exc: + print(f"::error file={path}::Unable to parse workflow: {exc}") + failures.append(path) + continue + for job in (doc.get("jobs") or {}).values(): + if not isinstance(job, dict): + continue + if job.get("name") != "CI Summary": + continue + steps = job.get("steps") or [] + if steps and all(is_placeholder(step) for step in steps): + failures.append(path) if failures: for path in failures:
PyYAMLis preinstalled onubuntu-24.04runners. Confirm this before you rely on it, or add an explicitpip install pyyamlstep.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.github/workflows/organization-pr-baseline.yml around lines 69 - 93, Replace the file-wide regex checks in the “Reject placeholder CI Summary gates” step with YAML parsing via PyYAML, confirming the runner dependency or installing it explicitly. Locate the job whose name is “CI Summary”, inspect only its steps’ run commands, and reject commands that are unconditional placeholders such as echo-only success, true, or exit 0 regardless of preceding setup commands. Report the workflow path and fail the step when such a CI Summary step is found.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In @.github/workflows/organization-pr-baseline.yml:
- Around line 29-35: Validate the combined runtime of the full-history checkout
and gitleaks fallback scan in the organization-pr-baseline workflow, especially
for workflow_call invocations with an empty BASE. Measure this on the largest
trial repository and adjust timeout-minutes or rollout scope based on the
observed runtime so legitimate scans do not fail due to the job timeout.
- Around line 95-106: The secret-scan logic in the pull-request workflow
duplicates secret-scan-reusable.yml and must have a single maintained
implementation. After the staged rollout, retire the older reusable workflow, or
update it to call the hardened scan implementation while preserving its range
selection, zero-SHA guard, commit verification fallback, and gitleaks flags.
- Around line 69-93: Replace the file-wide regex checks in the “Reject
placeholder CI Summary gates” step with YAML parsing via PyYAML, confirming the
runner dependency or installing it explicitly. Locate the job whose name is “CI
Summary”, inspect only its steps’ run commands, and reject commands that are
unconditional placeholders such as echo-only success, true, or exit 0 regardless
of preceding setup commands. Report the workflow path and fail the step when
such a CI Summary step is found.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 5ee87c90-1bfc-461f-8062-a89822f0cd27
📒 Files selected for processing (1)
.github/workflows/organization-pr-baseline.yml
The placeholder-gate check was a literal string match on `echo "CI checks passed"`. It missed four trivial variants (a `set -e` line before the echo, any other phrase, `exit 0`, and `true`) and fired on a workflow whose real gate was fine but which mentioned the phrase in an unrelated job. Parse the YAML and judge the gate job instead: a job is a fake gate when it neither aggregates other jobs through needs.<job>.result nor runs a command that can fail. Checked against all 216 workflow files in the organization — it flags the two known fake gates in meditation-service and healify-org, and nothing else. Add advisory checks for the debt the same audit measured: 561 of 657 external action references sit on mutable tags, 84 workflows carry no permissions block, 182 of 309 jobs have no timeout, and ten steps interpolate a secret straight into a run script. These annotate rather than fail so the workflow stays adoptable; a repository opts into enforcement with strict: true. Also stop cancelling in-progress runs on branch pushes, so a rapid series of commits cannot leave one unscanned behind a cancelled run; raise the default timeout to 20 minutes because the fallback secret scan walks all history; retry the tool downloads; and install PyYAML explicitly rather than assuming the runner image provides it.
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (1)
.github/workflows/organization-pr-baseline.yml (1)
236-250: 🩺 Stability & Availability | 🔵 Trivial | 💤 Low valueConfirm the fallback scan stays inside the job timeout.
The fallback at Line 249 walks the full history. The default timeout is 20 minutes. On a repository with a long history, gitleaks can exceed that budget, and the job fails with a timeout instead of a scan verdict. The input description at Line 28 documents the risk, but callers must set the value before the first run.
Consider printing the commit count before the fallback scan, so a timeout is attributable in the log.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.github/workflows/organization-pr-baseline.yml around lines 236 - 250, Update the fallback branch in the “Scan the pull-request range for committed secrets” step to print the repository’s commit count immediately before running the full-history gitleaks scan. Keep the existing fallback scan and timeout behavior unchanged, and make the log clearly identify the count as the scope of the fallback scan.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/organization-pr-baseline.yml:
- Around line 17-20: Update the pull_request trigger in the workflow
configuration to remove the paths filter so organization-pr-baseline runs for
every pull request, ensuring the existing gitleaks step executes regardless of
which files changed.
- Around line 123-144: Update real_commands to analyze each
shell-operator-separated command rather than matching INERT against the entire
line prefix. Treat a line as inert only when every command part is an explicitly
inert command, ensuring keywords such as echo, true, shift, and : match complete
command names and chained commands like echo ... && make verify are classified
as real.
- Around line 92-97: Update the workflow’s PyYAML setup and audit execution to
use an isolated Python environment, such as a virtual environment created with
the configured Python interpreter. Install PyYAML through that environment and
invoke the audit with the same interpreter, replacing the bare pip install and
system python3 usage in the “Ensure the YAML parser is present” step.
---
Nitpick comments:
In @.github/workflows/organization-pr-baseline.yml:
- Around line 236-250: Update the fallback branch in the “Scan the pull-request
range for committed secrets” step to print the repository’s commit count
immediately before running the full-history gitleaks scan. Keep the existing
fallback scan and timeout behavior unchanged, and make the log clearly identify
the count as the scope of the fallback scan.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 5f023f1f-06e1-42ed-a374-94843302de91
📒 Files selected for processing (1)
.github/workflows/organization-pr-baseline.yml
Three fixes from review. The paths filter meant the secret scan only ran when a pull request touched a workflow file, so a commit that put a credential in application code was never scanned. Drop the filter; the audit is fast and the repository is small. The inert-command check matched on line prefix, so `echo "verifying" && make verify` looked inert and a compliant gate could be reported as fake — a false positive in the blocking tier, which is the one thing that tier must not do. Split each line on shell operators and require every segment to be inert, and match command words whole so `truncate_logs` is no longer read as `true`. `exit 0` stays inert; any other exit can fail and counts as real. Installing PyYAML with a bare pip would have failed on Ubuntu 24.04, whose system Python is PEP 668 externally-managed. That path never ran because the image already ships PyYAML, which is precisely why it needed fixing rather than leaving as untested fallback. Use actions/setup-python, pinned by SHA. Regression cases now cover the compound-command false positive, whole-word matching, and the still-fake variants. Re-checked against all 216 workflow files in the organization: the same two fake gates, nothing else.
1 participant
Summary
A reusable, read-only pull-request baseline that Healify repositories can adopt before it is enforced through an organization ruleset.
Two tiers, and why
A 2026-08-12 audit of all 216 workflow files across 26 repositories measured the current state:
permissionsblocktimeout-minutesrunscriptCI Summarygates that run nothingFailing on the first four would red-line nearly every repository the day this is adopted, and a baseline nobody can turn on protects nothing. So they annotate. A repository opts into enforcement with
strict: trueonce it has cleaned up.The blocking tier is limited to what the fleet already passes: syntax, committed secrets, and fake merge gates.
Gate detection
The previous revision matched the literal string
echo "CI checks passed". That missed four trivial variants — aset -eline before the echo, any other phrase,exit 0, andtrue— and fired on a workflow whose gate was real but which mentioned the phrase in an unrelated job.This revision parses the YAML and judges the gate job: a job is a fake gate when it neither aggregates other jobs through
needs.<job>.resultnor runs a command that can fail.Verification
Run against the real corpus of all 216 workflow files, by extracting the audit script from this YAML so the tested code is the shipped code:
CI Summarygates inmeditation-serviceandhealify-orgare literallyrun: echo "CI checks passed".healify,healify-api,healify-agentcore,healify-web,dataroom, andtestflight-automation.a6a1f9e..adb558a(2 commits), and the advisory tier emitted 3 annotations againstsecret-scan-reusable.yml.Rollout note
Adopting this in
meditation-serviceandhealify-orgwill fail until theirCI Summarygates run real checks. That is the intended result, not a regression — with the enterprise ruleset makingCI Summarythe only required status, those two repositories currently have no effective gate on their default branch.Not in this change
secret-scan-reusable.ymloverlaps with the secret-scan step here and still floats onactions/checkout@v7. Retiring it or pointing it at this workflow is a rollout decision, kept out of this diff.Linear: https://linear.app/lifecycle-innovations/issue/HEA-7249