fix(skills): verify newly-added CI jobs pass, not just required checks#169
Closed
tend-agent wants to merge 1 commit intomainfrom
Closed
fix(skills): verify newly-added CI jobs pass, not just required checks#169tend-agent wants to merge 1 commit intomainfrom
tend-agent wants to merge 1 commit intomainfrom
Conversation
The CI monitoring loop polls only `--required` checks. When the bot adds a new CI job (which isn't yet required), it never notices if that job fails — it reports "CI passed" based on pre-existing required checks alone. Observed on PRQL/prql PR #5779: the bot added a `test-devcontainer` job that failed (uv not in PATH), but both the mention and review runs reported success because only `pre-commit.ci` (the sole required check) was polled. Add guidance to also verify any jobs the bot created or modified after the required-checks loop completes. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Evidence
Run IDs: 24116374057 (tend-mention), 24116452569 (tend-review)
Target repo: PRQL/prql, PR #5779
What happened: The mention run responded to a maintainer request by adding a
test-devcontainerjob totests.yaml. The new job failed (uvnot in PATH in the devcontainer CI environment). However, the bot's CI polling loop only checked--requiredchecks (pre-commit.ci), so it reported "CI passed." The subsequent review run also polled only required checks and concluded "no issues found." Both sessions incorrectly reasoned the new job "likely didn't run" due to path filters — it did run and failed.Root cause: The
running-in-ciskill's CI monitoring section instructs bots to pollgh pr checks --required. Newly added CI jobs aren't required, so their failures are invisible to this polling. The bot has no guidance to check its own newly-created jobs.Fix: Add a paragraph after the required-checks instructions telling bots to also verify any jobs they created or modified, using
gh pr checkswithout--required.Gate assessment
--requiredflag deterministically excludes new non-required jobs; will recur every time the bot adds a CI job🤖 Generated with Claude Code