Commit fecec8d
authored
workflows, ci_scripts: add automation logic from wheel_builder (#100)
* LICENSE-Apache-2.0: add
Some of the scripts we have ported from the wheel_builder repository are
unchanged, and the originals were licensed under Apache-2.0 terms. Add a
copy of that LICENSE file as LICENSE-Apache-2.0 for compliance purposes.
Signed-off-by: Trevor Gamblin <tgamblin@baylibre.com>
* ci_scripts: add audit.sh
Migrate the script from wheel_builder, with the following changes:
- Add new RISE copyright
- Include a short description
- Make it output a json report for further analysis
Signed-off-by: Trevor Gamblin <tgamblin@baylibre.com>
* ci_scripts: add check_versions.py
Migrate the check_versions.py script from wheel_builder. Much of the
script has been refactored with the following goals:
1. Use GitHub conventions (instead of GitLab)
2. Create GitHub-specific replacement functions where necessary
3. Simplify other functions where possible
4. Use more Pythonic styling
5. Remove comments where the code is self-documenting
More specifically:
- Replace usage of `glab` with the corresponding `gh` CLI
- Create a REPO variable pointing to "riseproject-dev/python-wheels",
and use it wherever the wheel_builder was referenced (in calls to
`glab`)
- Change configure_git_identity() to use a "github-actions[bot]" user
and corresponding noreply email
- Change generated branch names' prefix from "gitlab-ci" to
"github-actions"
- Remove some auth/login-related logic using tokens to generate
automated PRs, since the GitHub project allows us to enable this in
the repository settings (not programatically)
- Add dispatch_workflow() and create_upgrade_pr() functions for
triggering a desired workflow (e.g. for a "matplotlib" upgrade, the
script should trigger the "build-matplotlib.yml" and
"test-matplotlib.yml" workflows for the target version(s) if the
workflow scripts exist, or indicate otherwise if not possible
- Allow updating of default versions in workflow scripts when
auto-generating upgrade PRs.
- Add "Trigger" labels to auto-generated PRs for the new versions
detected.
- Importantly, limit the maximum number of versions to be targeted by an
auto-generated PR to three, so as not to create a runaway build queue.
- Remove most comments, since the code objectives are obvious after the
refactor
AI-Generated: Uses Claude Code Opus 4.7
Signed-off-by: Trevor Gamblin <tgamblin@baylibre.com>
* ci_scripts: check_patch.py: add
Migrate check_patch.py from the wheel_builder repository, adding a new
RISE copyright and updating the description.
Signed-off-by: Trevor Gamblin <tgamblin@baylibre.com>
* ci_scripts: audit_report: add
Create a new script for reviewing reports created by
ci_scripts/audit.sh, and opening/commenting on issues in GitHub to track
vulnerabilities identified by nightly pip-audit runs. The issues are
opened on a per-vulnerability basis with affected versions, but only if
an existing issue matching the same vulnerability and with the
"pip-audit" label isn't found. Previously-closed issues are ignored when
determining if a new one should be opened.
Also use GHA warning notations so that the nightly CI pipeline can
report a pass (instead of a failure) while still reporting a
vulnerability in the issue list.
AI-Generated: Uses Claude Code Opus 4.7
Signed-off-by: Trevor Gamblin <tgamblin@baylibre.com>
* ci_scripts: migrate deprecated.txt
Migrate the deprecated packages file from wheel_builder, adding a new
RISE copyright.
Signed-off-by: Trevor Gamblin <tgamblin@baylibre.com>
* ci_scripts: migrate packages.txt
Migrate the supported packages file from wheel_builder, adding a new
RISE copyright in the process.
Signed-off-by: Trevor Gamblin <tgamblin@baylibre.com>
* ci_scripts: add list-packages.sh
Migrate the script from wheel_builder, adding a new RISE copyright and a
short description in the process. Also make sure that it only tries to
install binary wheels, so that we don't try and build them from scratch
during simple checks.
Signed-off-by: Trevor Gamblin <tgamblin@baylibre.com>
* workflows: pr-trigger: add
Add a workflow for triggering workflows named "build-<package>.yml"
(and if present, "test-<package>.yml") with desired release tags for a
given package. For automated upgrade checks and manual changes, this
means that the tags only need to be specified in the PR description when
opened.
This is based on the process in wheel_builder where a child pipeline
would be generated automatically when a merge request was submitted with
one or more package version tags listed as e.g. "Trigger:
polars:py-1.42.1" in the MR description.
AI-Generated: Uses Claude Code Sonnet 4.7
Signed-off-by: Trevor Gamblin <tgamblin@baylibre.com>
* workflows: pr-checks: add
Provide a workflow for verifying that content added/modified in commits
meets basic requirements, i.e. no debug patches with "revertme", "revert
me", or "DO NOT MERGE" at the beginning. Also ensure that an
"Upstream-Status" line is provided in any carried patches (by running
check_patch.py), so that developers understand the patches' purpose.
This is based on a combination of the "check_commit_messages" and
"check_patches" jobs in wheel_builder's .gitlab-ci.yml script.
AI-Generated: Uses Claude Code Sonnet 4.7
Signed-off-by: Trevor Gamblin <tgamblin@baylibre.com>
* workflows: nightly: add
Provide a nightly CI workflow for checking:
1. Latest upstream package versions versus those available in the RISE
registry
2. Packages which have upstream riscv64 wheels (ready for deprecation in
our CI)
3. Check that packages install correctly in a riscv64 container
4. Run pip-audit to scan for vulnerabilities
This is based on the following jobs from wheel_builder's .gitlab-ci.yml
script:
- check_deprecated_packages
- check_installation
- check_versions
- pip_audit
Two notable differences compared to the original logic (in addition to
the GitLab CI -> GHA rework):
1. We use RISE's RISC-V Runners for the steps which would otherwise run
the setup-qemu action.
2. A step is added using the new audit_report.py script to file issues
when vulnerabilities are detected.
AI-Generated: Uses Claude Code Opus 4.7
Signed-off-by: Trevor Gamblin <tgamblin@baylibre.com>
* revertme: add pull_request to nightly workflow for testing
Signed-off-by: Trevor Gamblin <tgamblin@baylibre.com>
---------
Signed-off-by: Trevor Gamblin <tgamblin@baylibre.com>1 parent 42107e0 commit fecec8d
11 files changed
Lines changed: 1595 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
0 commit comments