Skip to content

ci: alert Slack when nightly e2e fails #44

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ilya-bogin-keenable merged 2 commits into from
Jun 16, 2026
Merged

ci: alert Slack when nightly e2e fails #44

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
dacb015
ci: alert Slack when nightly e2e fails
Bryunyon Jun 16, 2026
4431360
ci: alert Slack on cancelled e2e runs too, not just failures
Bryunyon Jun 16, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 33 additions & 0 deletions .github/workflows/e2e.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -132,3 +132,36 @@ jobs:
args=(-v -ra --color=yes)
[ -n "$MARKERS" ] && args+=(-m "$MARKERS")
uv run --project tests/e2e pytest tests/e2e "${args[@]}"

# Alert Slack when any e2e leg does not succeed. Runs only on the nightly
# schedule — manual dispatch is interactive, so the operator already sees the
# result. `result != 'success'` covers both failures and cancellations (the
# 30-min job timeout cancels rather than fails), so timed-out nightly runs
# still alert. Needs the SLACK_WEBHOOK_URL repo secret (a Slack Incoming
# Webhook URL; the target channel is fixed by the webhook). Self-skips if the
# secret is unset.
notify:
name: notify-slack-on-failure
needs: e2e
if: ${{ always() && github.event_name == 'schedule' && needs.e2e.result != 'success' }}
runs-on: ubuntu-latest
steps:
- name: Post failure to Slack
env:
SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}
RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
REPO: ${{ github.repository }}
run: |
if [ -z "$SLACK_WEBHOOK_URL" ]; then
echo "::warning::SLACK_WEBHOOK_URL not set — skipping Slack alert"
exit 0
fi
# Build the payload with jq so RUN_URL/REPO are JSON-escaped, never
# interpolated into the shell or the JSON string by hand.
payload="$(jq -n \
--arg repo "$REPO" \
--arg url "$RUN_URL" \
'{text: ("🔴 Nightly e2e failed for *" + $repo + "*\n<" + $url + "|View run>")}')"
curl --proto '=https' --tlsv1.2 -sSf --retry 3 \
-X POST -H 'Content-Type: application/json' \
-d "$payload" "$SLACK_WEBHOOK_URL"
Loading