Skip to content

ci: alert Slack when nightly e2e fails - #44

Merged
ilya-bogin-keenable merged 2 commits into
mainfrom
ci/slack-alert-on-e2e-failure
Jun 16, 2026
Merged

ci: alert Slack when nightly e2e fails#44
ilya-bogin-keenable merged 2 commits into
mainfrom
ci/slack-alert-on-e2e-failure

Conversation

@ilya-bogin-keenable

Copy link
Copy Markdown
Contributor

Что

Добавляет job notify в .github/workflows/e2e.yml, который пишет в Slack при падении ночного e2e-прогона.

Как работает

  • needs: e2e — ждёт всю матрицу (Linux/macOS/Windows; fail-fast: false).
  • if: failure() && github.event_name == 'schedule' — алерт только на ночном cron, не на ручном workflow_dispatch (там оператор и так видит результат).
  • Шлёт в Slack Incoming Webhook через секрет SLACK_WEBHOOK_URL (канал фиксируется самим webhook). Сообщение: статус + ссылка на конкретный run.
  • Payload собирается через jq -n --arg, значения JSON-экранируются.
  • Если секрет не задан — job не падает, пишет warning и скипается.

Setup

Секрет SLACK_WEBHOOK_URL уже добавлен в репозиторий. Webhook проверен (Slack ответил ok).

🤖 Generated with Claude Code

Add a notify job (needs: e2e, if: failure() && schedule) that posts to a
Slack Incoming Webhook (SLACK_WEBHOOK_URL secret) with the run link. Only
fires on the nightly schedule — manual dispatch is interactive. Self-skips
if the secret is unset; payload built with jq to JSON-escape values.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@qodo-code-review

qodo-code-review Bot commented Jun 16, 2026

Copy link
Copy Markdown

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📎 Requirement gaps (0) 🎨 UX issues (0) 🔗 Cross-repo conflicts (0) 📜 Skill insights (0)

Grey Divider


Remediation recommended

1. No cancel/timeout alert ✓ Resolved 🐞 Bug ☼ Reliability
Description
The new notify job only runs when failure() is true, so scheduled e2e runs that end as cancelled
(including common timeout cancellations) will not post a Slack alert even though the nightly run
didn’t succeed. This can silently miss the primary operational signal the job is meant to provide.
Code

.github/workflows/e2e.yml[143]

+    if: failure() && github.event_name == 'schedule'
Evidence
The workflow sets timeout-minutes: 30 for the e2e job, but the new notify job’s condition only
checks failure(), which does not include cancelled outcomes, so timeouts/cancellations won’t
trigger Slack alerts.

.github/workflows/e2e.yml[53-54]
.github/workflows/e2e.yml[140-144]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
The Slack notification job only runs on `failure()`, so it won't notify on cancelled runs (e.g., timeouts). This can cause missed alerts for nightly e2e runs that did not complete successfully.

### Issue Context
The e2e job has a 30-minute timeout, so timeout-driven cancellations are plausible and should be treated as "nightly did not succeed" for alerting purposes.

### Fix Focus Areas
- .github/workflows/e2e.yml[53-54]
- .github/workflows/e2e.yml[140-144]

### Suggested change
Update the `notify` job condition to run on any non-success result for scheduled runs, e.g.:

```yaml
notify:
 needs: e2e
 if: ${{ always() && github.event_name == 'schedule' && needs.e2e.result != 'success' }}
```

This covers both failures and cancellations while still restricting alerts to the nightly schedule.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


Grey Divider

Qodo Logo

@qodo-code-review

Copy link
Copy Markdown

PR Summary by Qodo

CI: Notify Slack on nightly E2E workflow failures
⚙️ Configuration changes ✨ Enhancement 🕐 10-20 Minutes

Grey Divider

Walkthroughs

Description
• Add a Slack notification job when the scheduled nightly E2E matrix fails.
• Gate alerts to cron runs only, avoiding noise on manual workflow_dispatch runs.
• Post a run link via Incoming Webhook, safely JSON-encoding payload with jq.
Diagram
graph TD
  A["GitHub Actions: E2E workflow"] --> B["e2e job (matrix)"] -->|"needs; only if failure() & schedule"| C["notify job"] --> D{{"Slack webhook"}}
Loading
High-Level Assessment

The following are alternative approaches to this PR:

1. Use slackapi/slack-github-action
  • ➕ Less custom shell scripting; purpose-built Slack payload support
  • ➕ Easier to extend to blocks/attachments and richer formatting
  • ➖ Adds a third-party action dependency and versioning/supply-chain considerations
  • ➖ Still requires secret management; not fundamentally simpler for a single message
2. Create a GitHub Issue / use GitHub Notifications instead of Slack
  • ➕ No external webhook/secrets required
  • ➕ Keeps alerts and history within GitHub
  • ➖ Doesn’t reach the team’s primary chat channel as quickly
  • ➖ Can create noisy issue churn unless carefully managed

Recommendation: The current approach (a small notify job using curl + jq with a webhook secret) is a good fit: minimal dependencies, runs only on scheduled failures, and avoids brittle manual JSON string interpolation. Consider adopting slackapi/slack-github-action only if you expect richer message formatting or more complex routing later.

Grey Divider

File Changes

Other (1)
e2e.yml Add Slack notify job for scheduled E2E failures +30/-0

Add Slack notify job for scheduled E2E failures

• Introduces a new 'notify' job that depends on the full E2E matrix and runs only when the workflow is triggered by 'schedule' and the matrix fails. The job posts a JSON-escaped payload (repo + run link) to a Slack Incoming Webhook from 'SLACK_WEBHOOK_URL', and self-skips with a warning if the secret is unset.

.github/workflows/e2e.yml


Grey Divider

Qodo Logo

failure() excludes cancellations, so a nightly run that hits the 30-min
job timeout (cancelled, not failed) would skip the Slack alert. Gate the
notify job on `needs.e2e.result != 'success'` under always() instead, so
timeouts/cancellations also fire the alert. Surfaced by Qodo review.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@ilya-bogin-keenable
ilya-bogin-keenable merged commit 5d5d96e into main Jun 16, 2026
12 checks passed
@ilya-bogin-keenable
ilya-bogin-keenable deleted the ci/slack-alert-on-e2e-failure branch June 16, 2026 08:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants