Skip to content

Commit fecec8d

Browse files
authored
workflows, ci_scripts: add automation logic from wheel_builder (#100)
* LICENSE-Apache-2.0: add Some of the scripts we have ported from the wheel_builder repository are unchanged, and the originals were licensed under Apache-2.0 terms. Add a copy of that LICENSE file as LICENSE-Apache-2.0 for compliance purposes. Signed-off-by: Trevor Gamblin <tgamblin@baylibre.com> * ci_scripts: add audit.sh Migrate the script from wheel_builder, with the following changes: - Add new RISE copyright - Include a short description - Make it output a json report for further analysis Signed-off-by: Trevor Gamblin <tgamblin@baylibre.com> * ci_scripts: add check_versions.py Migrate the check_versions.py script from wheel_builder. Much of the script has been refactored with the following goals: 1. Use GitHub conventions (instead of GitLab) 2. Create GitHub-specific replacement functions where necessary 3. Simplify other functions where possible 4. Use more Pythonic styling 5. Remove comments where the code is self-documenting More specifically: - Replace usage of `glab` with the corresponding `gh` CLI - Create a REPO variable pointing to "riseproject-dev/python-wheels", and use it wherever the wheel_builder was referenced (in calls to `glab`) - Change configure_git_identity() to use a "github-actions[bot]" user and corresponding noreply email - Change generated branch names' prefix from "gitlab-ci" to "github-actions" - Remove some auth/login-related logic using tokens to generate automated PRs, since the GitHub project allows us to enable this in the repository settings (not programatically) - Add dispatch_workflow() and create_upgrade_pr() functions for triggering a desired workflow (e.g. for a "matplotlib" upgrade, the script should trigger the "build-matplotlib.yml" and "test-matplotlib.yml" workflows for the target version(s) if the workflow scripts exist, or indicate otherwise if not possible - Allow updating of default versions in workflow scripts when auto-generating upgrade PRs. - Add "Trigger" labels to auto-generated PRs for the new versions detected. - Importantly, limit the maximum number of versions to be targeted by an auto-generated PR to three, so as not to create a runaway build queue. - Remove most comments, since the code objectives are obvious after the refactor AI-Generated: Uses Claude Code Opus 4.7 Signed-off-by: Trevor Gamblin <tgamblin@baylibre.com> * ci_scripts: check_patch.py: add Migrate check_patch.py from the wheel_builder repository, adding a new RISE copyright and updating the description. Signed-off-by: Trevor Gamblin <tgamblin@baylibre.com> * ci_scripts: audit_report: add Create a new script for reviewing reports created by ci_scripts/audit.sh, and opening/commenting on issues in GitHub to track vulnerabilities identified by nightly pip-audit runs. The issues are opened on a per-vulnerability basis with affected versions, but only if an existing issue matching the same vulnerability and with the "pip-audit" label isn't found. Previously-closed issues are ignored when determining if a new one should be opened. Also use GHA warning notations so that the nightly CI pipeline can report a pass (instead of a failure) while still reporting a vulnerability in the issue list. AI-Generated: Uses Claude Code Opus 4.7 Signed-off-by: Trevor Gamblin <tgamblin@baylibre.com> * ci_scripts: migrate deprecated.txt Migrate the deprecated packages file from wheel_builder, adding a new RISE copyright. Signed-off-by: Trevor Gamblin <tgamblin@baylibre.com> * ci_scripts: migrate packages.txt Migrate the supported packages file from wheel_builder, adding a new RISE copyright in the process. Signed-off-by: Trevor Gamblin <tgamblin@baylibre.com> * ci_scripts: add list-packages.sh Migrate the script from wheel_builder, adding a new RISE copyright and a short description in the process. Also make sure that it only tries to install binary wheels, so that we don't try and build them from scratch during simple checks. Signed-off-by: Trevor Gamblin <tgamblin@baylibre.com> * workflows: pr-trigger: add Add a workflow for triggering workflows named "build-<package>.yml" (and if present, "test-<package>.yml") with desired release tags for a given package. For automated upgrade checks and manual changes, this means that the tags only need to be specified in the PR description when opened. This is based on the process in wheel_builder where a child pipeline would be generated automatically when a merge request was submitted with one or more package version tags listed as e.g. "Trigger: polars:py-1.42.1" in the MR description. AI-Generated: Uses Claude Code Sonnet 4.7 Signed-off-by: Trevor Gamblin <tgamblin@baylibre.com> * workflows: pr-checks: add Provide a workflow for verifying that content added/modified in commits meets basic requirements, i.e. no debug patches with "revertme", "revert me", or "DO NOT MERGE" at the beginning. Also ensure that an "Upstream-Status" line is provided in any carried patches (by running check_patch.py), so that developers understand the patches' purpose. This is based on a combination of the "check_commit_messages" and "check_patches" jobs in wheel_builder's .gitlab-ci.yml script. AI-Generated: Uses Claude Code Sonnet 4.7 Signed-off-by: Trevor Gamblin <tgamblin@baylibre.com> * workflows: nightly: add Provide a nightly CI workflow for checking: 1. Latest upstream package versions versus those available in the RISE registry 2. Packages which have upstream riscv64 wheels (ready for deprecation in our CI) 3. Check that packages install correctly in a riscv64 container 4. Run pip-audit to scan for vulnerabilities This is based on the following jobs from wheel_builder's .gitlab-ci.yml script: - check_deprecated_packages - check_installation - check_versions - pip_audit Two notable differences compared to the original logic (in addition to the GitLab CI -> GHA rework): 1. We use RISE's RISC-V Runners for the steps which would otherwise run the setup-qemu action. 2. A step is added using the new audit_report.py script to file issues when vulnerabilities are detected. AI-Generated: Uses Claude Code Opus 4.7 Signed-off-by: Trevor Gamblin <tgamblin@baylibre.com> * revertme: add pull_request to nightly workflow for testing Signed-off-by: Trevor Gamblin <tgamblin@baylibre.com> --------- Signed-off-by: Trevor Gamblin <tgamblin@baylibre.com>
1 parent 42107e0 commit fecec8d

11 files changed

Lines changed: 1595 additions & 0 deletions

File tree

.github/workflows/nightly.yml

Lines changed: 101 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,101 @@
1+
# SPDX-FileCopyrightText: 2026 The RISE Project
2+
# SPDX-License-Identifier: MIT
3+
4+
name: Run nightly package version and status checks
5+
6+
on:
7+
pull_request:
8+
types: [opened, edited, synchronize, reopened]
9+
schedule:
10+
- cron: '0 2 * * *'
11+
workflow_dispatch:
12+
13+
permissions:
14+
contents: write
15+
pull-requests: write
16+
actions: write
17+
issues: read
18+
19+
env:
20+
PIP_EXTRA_INDEX_URL: https://gitlab.com/api/v4/projects/56254198/packages/pypi/simple
21+
22+
jobs:
23+
check_versions:
24+
runs-on: ubuntu-latest
25+
steps:
26+
- uses: actions/checkout@v4
27+
with:
28+
fetch-depth: 0
29+
30+
- uses: actions/setup-python@v5
31+
with:
32+
python-version: '3'
33+
34+
- name: Install Python dependencies
35+
run: pip install requests packaging
36+
37+
- name: Check versions and open PRs
38+
env:
39+
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
40+
run: python3 ci_scripts/check_versions.py --summary --create-prs
41+
42+
check_deprecated_packages:
43+
runs-on: ubuntu-latest
44+
steps:
45+
- uses: actions/checkout@v4
46+
47+
- name: Install curl and jq
48+
run: sudo apt-get update -qq && sudo apt-get install -qq -y curl jq
49+
50+
- name: Verify riscv64 wheels still published
51+
run: |
52+
ret=0
53+
while IFS= read -r pkg; do
54+
[[ -z "$pkg" || "$pkg" =~ ^[[:space:]]*# ]] && continue
55+
56+
if ! output=$(curl -sf "https://pypi.org/pypi/${pkg}/json"); then
57+
echo "Failed to fetch $pkg, skipping..."
58+
continue
59+
fi
60+
61+
if ! filenames=$(echo "$output" | jq -r '.urls[].filename'); then
62+
echo "Failed to parse JSON for $pkg, skipping..."
63+
continue
64+
fi
65+
66+
if ! echo "$filenames" | grep -qi riscv; then
67+
echo "$pkg does not distribute riscv64 wheel anymore!"
68+
ret=1
69+
fi
70+
done < ci_scripts/deprecated.txt
71+
exit $ret
72+
73+
check_installation:
74+
runs-on: ubuntu-24.04-riscv
75+
steps:
76+
- uses: actions/checkout@v4
77+
78+
- name: Install packages inside riscv64 container
79+
working-directory: ci_scripts
80+
run: |
81+
sed -Ei 's/^scipy.*$//' packages.txt
82+
sed -Ei 's/^patchelf.*$//' packages.txt
83+
./list-packages.sh python-wheels
84+
85+
pip_audit:
86+
runs-on: ubuntu-24.04-riscv
87+
continue-on-error: true
88+
permissions:
89+
contents: read
90+
issues: write
91+
steps:
92+
- uses: actions/checkout@v4
93+
94+
- name: Run pip-audit
95+
working-directory: ci_scripts
96+
run: ./audit.sh python-wheels
97+
98+
- name: Report vulnerabilities and open issues
99+
env:
100+
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
101+
run: python3 ci_scripts/audit_report.py ci_scripts/audit-report.json

.github/workflows/pr-checks.yml

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
# SPDX-FileCopyrightText: 2026 The RISE Project
2+
# SPDX-License-Identifier: MIT
3+
4+
name: PR verification checks
5+
6+
on:
7+
pull_request:
8+
9+
jobs:
10+
check_commit_messages:
11+
runs-on: ubuntu-latest
12+
steps:
13+
- uses: actions/checkout@v4
14+
with:
15+
fetch-depth: 0
16+
17+
- name: Reject revertme / DO NOT MERGE commits
18+
env:
19+
BASE_SHA: ${{ github.event.pull_request.base.sha }}
20+
HEAD_SHA: ${{ github.event.pull_request.head.sha }}
21+
run: |
22+
set -euo pipefail
23+
range="$BASE_SHA..$HEAD_SHA"
24+
if git log --pretty=format:%s "$range" | grep -i -e '^revertme' -e '^revert me'; then
25+
echo "::error::Merge request contains at least a commit starting with a 'revert me' tag. Fix it before merging."
26+
exit 1
27+
fi
28+
if git log --pretty=format:%s "$range" | grep -i -e '^DO NOT MERGE'; then
29+
echo "::error::Merge request contains at least a commit starting with a 'DO NOT MERGE' tag. Review it."
30+
exit 1
31+
fi
32+
33+
check_patches:
34+
runs-on: ubuntu-latest
35+
steps:
36+
- uses: actions/checkout@v4
37+
with:
38+
fetch-depth: 0
39+
40+
- uses: actions/setup-python@v5
41+
with:
42+
python-version: '3'
43+
44+
- name: Validate Upstream-Status in added/modified patches
45+
env:
46+
BASE_SHA: ${{ github.event.pull_request.base.sha }}
47+
HEAD_SHA: ${{ github.event.pull_request.head.sha }}
48+
run: python ci_scripts/check_patch.py "$BASE_SHA" "$HEAD_SHA"

.github/workflows/pr-trigger.yml

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
# SPDX-FileCopyrightText: 2026 The RISE Project
2+
# SPDX-License-Identifier: MIT
3+
4+
name: PR package build trigger
5+
6+
on:
7+
pull_request:
8+
types: [opened, edited, synchronize, reopened]
9+
10+
permissions:
11+
contents: read
12+
actions: write
13+
pull-requests: read
14+
15+
jobs:
16+
dispatch-triggers:
17+
runs-on: ubuntu-latest
18+
if: contains(github.event.pull_request.body, 'Trigger:')
19+
steps:
20+
- uses: actions/checkout@v4
21+
with:
22+
package_version: ${{ github.event.pull_request.head.sha }}
23+
24+
- name: Dispatch build/test workflows for Trigger directives
25+
env:
26+
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
27+
PR_BODY: ${{ github.event.pull_request.body }}
28+
HEAD_REF: ${{ github.event.pull_request.head.ref }}
29+
HEAD_REPO: ${{ github.event.pull_request.head.repo.full_name }}
30+
BASE_REPO: ${{ github.repository }}
31+
run: |
32+
set -euo pipefail
33+
34+
if [ "$HEAD_REPO" != "$BASE_REPO" ]; then
35+
echo "PR from fork ($HEAD_REPO); cannot dispatch workflows against fork ref. Skipping."
36+
exit 0
37+
fi
38+
39+
triggers=$(printf '%s\n' "$PR_BODY" | grep -oE '^Trigger:[[:space:]]*[^[:space:]]+' || true)
40+
41+
if [ -z "$triggers" ]; then
42+
echo "No Trigger: directives found."
43+
exit 0
44+
fi
45+
46+
printf '%s\n' "$triggers" | while IFS= read -r line; do
47+
directive=${line#Trigger:}
48+
directive=${directive# }
49+
package_name=${directive%:*}
50+
package_version=${directive#*:}
51+
52+
if [ -z "$package_name" ] || [ -z "$package_version" ] || [ "$package_name" = "$package_version" ]; then
53+
echo "Skipping malformed directive: $line"
54+
continue
55+
fi
56+
57+
# Normalize: build workflows prepend `v` to the version themselves
58+
# (e.g. `ref: v${{ env.NUMPY_VERSION }}`). Strip any leading v/V
59+
# in the directive so `Trigger: numpy:v2.5.1` and
60+
# `Trigger: numpy:2.5.1` behave identically.
61+
package_version=${package_version#[vV]}
62+
63+
build_wf=".github/workflows/build-${package_name}.yml"
64+
test_wf=".github/workflows/test-${package_name}.yml"
65+
66+
if [ ! -f "$build_wf" ]; then
67+
echo "No build workflow for $package_name at $build_wf; skipping."
68+
continue
69+
fi
70+
71+
echo "Dispatching build-${package_name}.yml on $HEAD_REF with version=$package_version"
72+
gh workflow run "build-${package_name}.yml" --ref "$HEAD_REF" -f "version=$package_version"
73+
74+
if [ -f "$test_wf" ]; then
75+
echo "Dispatching test-${package_name}.yml on $HEAD_REF with version=$package_version"
76+
gh workflow run "test-${package_name}.yml" --ref "$HEAD_REF" -f "version=$package_version"
77+
fi
78+
done

0 commit comments

Comments
 (0)