Scheduled drift detection: weekly full-suite re-run against fresh resolves - #107
Conversation
- weekly re-run of the full test workflow against a fresh dependency resolve of unchanged main; this repo has NO push-to-main CI, so between PRs nothing else proves main still works - failures file ONE deduped drift-labeled issue; green runs close it - run_tests.yml gains workflow_call; setup-uv unpinned from the ancient 0.6.5; the per-PR cache-nuking block removed (every run cold- downloaded the multi-GB torch/bayesflow/keras/sbi stack) with uv caching enabled instead
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
🚧 Files skipped from review as they are similar to previous changes (1)
📝 WalkthroughWalkthroughChangesDrift monitoring
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant Trigger as Schedule or manual dispatch
participant Drift as Drift workflow
participant Tests as Reusable test workflow
participant Issues as GitHub Issues
Trigger->>Drift: Start workflow with optional force_fail
Drift->>Tests: Run reusable test workflow
Tests-->>Drift: Return test and rehearsal results
Drift->>Issues: Close, update, or create deduplicated drift issue
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Pull request overview
Adds scheduled “drift detection” to continuously validate main against fresh dependency resolves by reusing the existing test workflow, and improves CI performance by enabling uv caching and removing the prior cache-nuking/reinstall behavior.
Changes:
- Add a new scheduled/manual
drift.ymlworkflow that runs the full test suite and creates/updates/closes a single dedupeddrift-labeled issue based on results. - Make
run_tests.ymlreusable viaworkflow_callso it can be invoked bydrift.yml. - Update
run_tests.ymlto usesetup-uvwith caching enabled and remove the cache-clearing +uv sync --reinstallbehavior.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| .github/workflows/run_tests.yml | Adds workflow_call trigger and enables uv caching / removes forced reinstall for the main test job. |
| .github/workflows/drift.yml | New scheduled workflow that calls the test workflow and manages a deduped drift-tracking issue. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| tests: | ||
| uses: ./.github/workflows/run_tests.yml | ||
|
|
| - name: Install uv | ||
| uses: astral-sh/setup-uv@v7 | ||
| with: | ||
| version: "0.6.5" | ||
| # enable-cache: true | ||
| # cache-dependency-glob: "pyproject.toml pdm.lock" | ||
|
|
||
| - name: Clear all caches | ||
| run: | | ||
| rm -rf ~/.cache/pip | ||
| rm -rf ~/.cache/uv | ||
| rm -rf ~/.cache/conda | ||
| rm -rf ~/.cache/npm | ||
|
|
||
| version: "latest" | ||
| enable-cache: true | ||
| cache-dependency-glob: "pyproject.toml" |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/drift.yml:
- Around line 42-59: Update the FAILED_JOBS decision logic in the drift workflow
so an existing issue is closed only when RESULT_TESTS equals success. Treat
cancelled, skipped, and any other non-success test results as inconclusive by
keeping the issue open, while preserving the existing force-fail handling.
- Around line 26-31: Add a job-level concurrency group to the report job in
drift.yml, with cancel-in-progress set to false, so overlapping scheduled or
manual runs serialize the issue list-and-create sequence used by the report
steps.
- Around line 23-24: Update the drift workflow’s tests and reporting jobs so
they only execute when the workflow ref is the repository’s default branch,
including manually dispatched runs with a supplied ref. Apply the same
default-branch condition to the job that uses needs.tests.result for shared
drift issue updates, while preserving the existing test and reporting behavior
on the default branch.
- Around line 18-20: Restrict the issues: write permission to the report job in
the workflow, and remove it from the top-level permissions inherited by the
reusable tests job. Keep contents: read at the appropriate shared scope and
ensure report retains issues: write.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: b6a9efdc-aa3b-4c12-9ce9-1ca446901527
📒 Files selected for processing (2)
.github/workflows/drift.yml.github/workflows/run_tests.yml
| tests: | ||
| uses: ./.github/workflows/run_tests.yml |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== files =="
git ls-files .github/workflows || true
echo "== drift workflow =="
sed -n '1,220p' .github/workflows/drift.yml 2>/dev/null || true
echo "== tests workflow outline =="
if [ -f .github/workflows/run_tests.yml ]; then
wc -l .github/workflows/run_tests.yml
ast-grep outline .github/workflows/run_tests.yml --view expanded || true
sed -n '1,260p' .github/workflows/run_tests.yml
fi
echo "== references to issue/drift/main/reusable workflow =="
rg -n "issue|drift|main|workflow_dispatch|uses: \./\.github/workflows/run_tests|needs:|result|close|open|gh" .github/workflows/. 2>/dev/null || trueRepository: lnccbrown/LANfactory
Length of output: 8701
🌐 Web query:
GitHub reusable workflows github.ref default branch input ref workflow_dispatch workflow_call
💡 Result:
In GitHub Actions, the github.ref context and the calling mechanism for reusable workflows have specific behaviors you should be aware of regarding branches, workflow dispatch, and workflow calls. ### Using References in Reusable Workflows You cannot use expressions or context variables (like ${{ github.ref }} or ${{ github.sha }}) inside the uses keyword when calling a reusable workflow [1][2][3]. GitHub Actions requires a static reference—a branch name, tag, or commit SHA—to parse the workflow file at execution time [2][3]. * Same Repository: If your reusable workflow is in the same repository, you should omit the @{ref} suffix entirely (e.g., uses: ./.github/workflows/my-workflow.yml) [1][2][3]. When you omit this, GitHub automatically executes the workflow version from the same commit as the caller workflow [1][2]. * External Repositories: If calling a workflow from another repository, you must hardcode the version (e.g., uses: octo-org/repo/.github/workflows/my-workflow.yml@v1) [1][3]. ### The github Context in Reusable Workflows When a reusable workflow is triggered, the github context (including github.ref) is associated with the caller workflow, not the called reusable workflow [4][5]. This means the reusable workflow does not inherently "know" which specific branch or tag was used to call it if it was referenced by a tag or branch name [5]. If your reusable workflow needs to act on a specific branch or tag (e.g., for actions/checkout), you must explicitly pass that reference as an input [5][3]: 1. Define an input in the reusable workflow's on.workflow_call.inputs section [6][4]. 2. Pass the value from the caller using ${{ github.ref }} or ${{ github.ref_name }} inside the jobs.<job_id>.with block [4][5]. ### Workflow Dispatch vs. Workflow Call * Workflow Dispatch: The workflow_dispatch event allows you to trigger a workflow manually [6][7]. It is distinct from workflow_call (reusable workflows) and only receives events when the workflow file exists on the default branch [6]. * Inputs: While both workflow_dispatch and workflow_call support inputs, they serve different purposes [6][4]. workflow_call allows one workflow to use another as a component, whereas workflow_dispatch is typically used for on-demand execution [6][7]. Remember that the inputs context inside a reusable workflow is populated only by the values defined in its own on.workflow_call configuration [4][8]. Attempting to access inputs in the caller workflow's job-level if conditions will fail, as those inputs exist only within the scope of the called workflow [8].
Citations:
- 1: https://docs.github.com/en/actions/how-tos/reuse-automations/reuse-workflows
- 2: https://stackoverflow.com/questions/74945550/using-github-ref-when-triggering-a-reusable-workflow
- 3: https://stackoverflow.com/questions/69737232/github-context-variables-not-evaluating-for-reusable-workflow-reference
- 4: https://docs.github.com/en/actions/reference/workflows-and-actions/contexts
- 5: make reference accessible in reusable workflow actions/toolkit#1264
- 6: https://docs.github.com/actions/using-workflows/workflow-syntax-for-github-actions
- 7: https://github.com/marketplace/actions/workflow-dispatch
- 8: https://latchkey.dev/learn/github-actions/reusable-workflow-if-inputs-not-available-in-ci
🌐 Web query:
site:docs.github.com GitHub Actions reusable workflows workflow_dispatch branch workflow_call ref input github.ref default branch
💡 Result:
GitHub Actions utilizes specific mechanisms for triggering, referencing, and passing data to workflows, with distinct behaviors for reusable and manually dispatched workflows. 1. Reusable Workflows (workflow_call) Reusable workflows are defined using the workflow_call trigger [1][2]. They can be called from another workflow (the caller) by referencing their file path and a specific version (ref) [1][3]. - Referencing: You can reference a reusable workflow using the syntax {owner}/{repo}/.github/workflows/{filename}@{ref} [1][4]. The {ref} can be a branch name, a release tag, or a commit SHA [4][5]. - Context and Inputs: When a reusable workflow is triggered, the github context is associated with the caller workflow, not the called workflow [5][6]. Inputs are defined in the called workflow under on.workflow_call.inputs [2] and passed from the caller workflow using the with keyword in a job [1][2]. - Repository Context: Actions within a reusable workflow run as if they were part of the caller workflow; for example, actions/checkout will check out the caller repository's code, not the reusable workflow's repository [3]. 2. Manually Triggered Workflows (workflow_dispatch) The workflow_dispatch trigger allows workflows to be run manually [7]. - Default Branch Requirement: Traditionally, for a workflow to be triggered via workflow_dispatch, the workflow file must exist on the repository's default branch [8][7]. - Branch Execution: While the file must reside on the default branch to be visible and available for manual triggering, you can execute the workflow on a different branch by specifying the ref (e.g., using the --ref flag in the GitHub CLI: gh workflow run WORKFLOW --ref BRANCH) [7]. - Inputs: Inputs for workflow_dispatch are defined in the workflow file and accessed within the workflow via the inputs context or github.event.inputs [8][2][9]. 3. Key Considerations for github.ref - github.ref: This context variable represents the fully-formed ref (e.g., refs/heads/<branch_name> or refs/tags/<tag_name>) of the branch or tag that triggered the workflow [10]. In the context of reusable workflows, because the github context is inherited from the caller [5][6], github.ref will point to the branch or tag of the caller repository that initiated the run, not the branch of the repository hosting the reusable workflow [10].
Citations:
- 1: https://docs.github.com/en/actions/how-tos/reuse-automations/reuse-workflows
- 2: https://docs.github.com/enterprise-cloud@latest/actions/using-workflows/workflow-syntax-for-github-actions
- 3: https://docs.github.com/en/actions/concepts/workflows-and-actions/reusing-workflow-configurations
- 4: https://docs.github.com/en/enterprise-cloud@latest/actions/how-tos/reuse-automations/reuse-workflows
- 5: https://docs.github.com/en/enterprise-server@3.13/actions/reference/reusable-workflows-reference
- 6: https://docs.github.com/en/actions/reference/workflows-and-actions/reusing-workflow-configurations
- 7: https://docs.github.com/en/actions/how-tos/manage-workflow-runs/manually-run-a-workflow
- 8: https://docs.github.com/actions/using-workflows/triggering-a-workflow
- 9: https://docs.github.com/en/actions/reference/workflows-and-actions/contexts
- 10: https://docs.github.com/en/actions/reference/workflows-and-actions/variables
Restrict drift reporting to the default branch.
workflow_dispatch can run this call from a non-default branch when a ref is supplied. The tests job then runs that branch, but report: always() uses needs.tests.result to create or close the shared drift issue, so a passing branch can close an open issue and a failing branch can create a false drift issue. Drive both jobs from the default branch before using the result for issue updates.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.github/workflows/drift.yml around lines 23 - 24, Update the drift
workflow’s tests and reporting jobs so they only execute when the workflow ref
is the repository’s default branch, including manually dispatched runs with a
supplied ref. Apply the same default-branch condition to the job that uses
needs.tests.result for shared drift issue updates, while preserving the existing
test and reporting behavior on the default branch.
…ebooks Addresses the review on #107. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.
Suppressed comments (3)
.github/workflows/drift.yml:28
run_tests.ymlusessecrets.CODECOV_TOKEN(Codecov upload). When calling it fromdrift.yml, that secret is not passed unless explicitly forwarded, which can cause the drift run to fail or skip coverage upload unexpectedly. Pass the token explicitly (or usesecrets: inheritif preferred).
# third-party test code should not carry it.
permissions:
contents: read
uses: ./.github/workflows/run_tests.yml
.github/workflows/drift.yml:47
inputs.force_failis referenced in theenv:block, butinputscontext only exists forworkflow_dispatch/workflow_call. On the scheduled trigger this can fail workflow expression evaluation (or set an unexpected empty value), breaking drift runs. Usegithub.event.inputs.force_failwith a default instead so schedule runs work.
env:
GH_TOKEN: ${{ github.token }}
RESULT_TESTS: ${{ needs.tests.result }}
FORCE_FAIL: ${{ inputs.force_fail }}
run: |
.github/workflows/run_tests.yml:5
run_tests.ymlis now a reusable workflow (workflow_call), but it referencessecrets.CODECOV_TOKEN. Reusable workflows must explicitly declare which secrets they accept; otherwise callers cannot pass them and the Codecov step may fail or behave differently when invoked fromdrift.yml. DeclareCODECOV_TOKENunderon.workflow_call.secrets.
on:
pull_request:
workflow_call: # Called by drift.yml (scheduled drift detection)
|
Thanks — I worked through all six findings. Two were real and are now fixed; the rest I've assessed below. Fixed
Also applied (not flagged): Applied, with a caveat about the stated severity
Not applied, with reasons
On removing the cache-nuking block — I checked the history since it's the kind of thing that looks load-bearing. |
|
The three red The failing step is Identical on both, against unmodified #103 is already the fix and has been open since 31 July: it adds Three interactions worth settling before either merges, since #103 and this PR edit the same region of
Suggested order: #103 first, then I rebase this on top, resolve |
|
Copilot's re-review of the current head produced no new comments; its three suppressed notes are the only open items. Two are the CODECOV_TOKEN point I addressed above (fail-soft: The third is new and worth answering properly, because it concerns the one path no rehearsal can exercise:
Checked, and it doesn't hold. The evidence is in the sibling repo, on exactly this construct. on:
pull_request:
workflow_call:
inputs: { run_slow_tests: ..., test_args: ... }
jobs:
run_tests:
env:
HSSM_TEST_ARGS: ${{ inputs.test_args }} # <- env block, same as FORCE_FAIL
steps:
- name: Run fast tests
if: ${{ !inputs.run_slow_tests }}On a That maps directly onto the drift case: on the weekly cron, I considered switching to Worth being explicit that this is a genuine gap in what the rehearsals prove: every validation so far has been via |
Part of the ecosystem self-healing rollout (aggregation layer: lnccbrown/HSSMSpine#35; siblings: lnccbrown/HSSM#1143, lnccbrown/ssm-simulators#318).
drift.yml: weekly scheduled re-run of the full test workflow viaworkflow_callagainst a fresh PyPI resolve of unchanged main — this repo has no push-to-main CI at all, so between PRs nothing else proves main still worksdrift-labeled issue; green runs close it;force_faildispatch input rehearses the pathrun_tests.yml:workflow_call:trigger; setup-uv unpinned from the ancient0.6.5; the per-PR cache-nuking +--reinstallblock removed (every PR cold-downloaded the multi-GB torch/bayesflow/keras/sbi stack), uv caching enabled instead🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Chores