Skip to content

feat(canaries): add SRE Agent notification workflow to canary alerts - #2249

Open
ujwalsah wants to merge 1 commit into
masterfrom
feat/canary-alerts-sre-agent
Open

feat(canaries): add SRE Agent notification workflow to canary alerts#2249
ujwalsah wants to merge 1 commit into
masterfrom
feat/canary-alerts-sre-agent

Conversation

@ujwalsah

Copy link
Copy Markdown
Contributor

Summary

  • Adds a New Relic notification channel and workflow to route canary alert policy incidents to the SRE Agent for automated root cause analysis
  • When a canary alert fires (comparing infra agent versions), the SRE Agent receives full alert context, analyzes historical trends, and writes findings back into the alert issue
  • Follows the same pattern used by the Virtuoso team (source.datanerd.us/virtuoso/install-atlantis/pull/525)

Test plan

  • Run terraform plan in test/canaries/alerts/ to validate configuration
  • Trigger a manual workflow dispatch via ci_canary_alerts_provision.yml
  • Verify the workflow appears in NR UI under Alerts > Workflows
  • Confirm the SRE Agent destination receives notifications and writes analysis back into alert issues

Route canary alert policy incidents to the NR SRE Agent for automated
root cause analysis. When a canary alert fires, the workflow sends alert
context to the SRE Agent which analyzes historical trends and writes its
findings back into the alert issue.
@ujwalsah
ujwalsah requested a review from a team as a code owner May 25, 2026 09:34
@coveralls

Copy link
Copy Markdown

Coverage Report for CI Build 26393732020

Coverage decreased (-0.1%) to 53.475%

Details

  • Coverage decreased (-0.1%) from the base build.
  • Patch coverage: No coverable lines changed in this PR.
  • 166 coverage regressions across 7 files.

Uncovered Changes

No uncovered changes found.

Coverage Regressions

166 previously-covered lines in 7 files lost coverage.

File Lines Losing Coverage Coverage
pkg/config/config.go 61 71.25%
pkg/sysinfo/cloud/cloud_oci.go 46 21.37%
pkg/integrations/v4/logs/cfg.go 44 86.41%
pkg/integrations/legacy/runner.go 6 77.96%
pkg/license/license.go 6 26.09%
pkg/config/defaults.go 2 66.67%
internal/agent/event_sender.go 1 78.11%

Coverage Stats

Coverage Status
Relevant Lines: 27119
Covered Lines: 14502
Line Coverage: 53.48%
Coverage Strength: 20.65 hits per line

💛 - Coveralls

@muruganishekar muruganishekar left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

resource "newrelic_notification_channel" "sre_agent_channel" {
name = "Canary Alerts SRE Agent Channel"
account_id = var.account_id
destination_id = "0b7bf32c-f336-442e-8be4-42e5f3eda718"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we use it from vars instead of directly using it ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants