Make webhook health thresholds config-controlled instead of compile-time constants

[jiashuoz]

Problem

Every tunable in the webhook health/breaker path is a compile-time constant. Changing any of them requires a code change → merge → OSS release → promote PR → prod deploy. That's a full release cycle to adjust a number.

constant	value	location
WarnThreshold	5	internal/identity/webhooks.go:594
WarnWindow	24h	:595
AutoDisableThreshold	10	:578
AutoDisableWindow	72h	:579
WarnSweepMaxPerTick	100	:667
DisableSweepMaxPerTick	100	:659
MaxDisabledSnoozes	24	internal/webhookdelivery/worker.go:58
disabledSnooze	1h	:46

The current values are fine and are staying — this is about the mechanism, not the numbers.

Why it matters

These are volume-dependent, and e2a is self-hostable. A count-based threshold that's right at one deployment's traffic is wrong at 100× or 1/100×. WarnThreshold is the clearest case: a webhook receiving 3 events/day can never accumulate 5 failures in 24h, so it would never warn no matter how thoroughly broken. One compiled value cannot serve every operator.

Some of these are incident-response levers. WarnSweepMaxPerTick and DisableSweepMaxPerTick exist to stop a systemic e2a-side failure mass-mailing the customer base. During an actual incident you'd want to turn those down now, not ship a release.

Suggested shape

A webhooks: config block (these are delivery/breaker settings, not notification settings, so they don't belong under notifications:), with the same optional + env-override pattern the existing config uses:

webhooks:
  warn_threshold: 5
  warn_window: 24h
  auto_disable_threshold: 10
  auto_disable_window: 72h
  sweep_max_per_tick: 100
  disabled_snooze_cap: 24

Unset must keep today's compiled defaults, so self-hosts and existing deployments are unchanged.

Validation is load-bearing here, more than usual

A wrong value in this block can auto-disable healthy webhooks fleet-wide — and auto-disable is lossy forwards (events published while disabled are never queued for the endpoint and cannot be replayed to it). So the config loader should reject rather than coerce:

• thresholds < 1 (a threshold of 0 disables every webhook with any delivery history)
• windows <= 0, or shorter than the retry envelope (29h21m) for auto_disable_window — terminal failures cannot accumulate faster than that, so a shorter window silently makes the breaker unreachable
• per-tick caps < 1 (silently disables the sweep)
• disabled_snooze_cap < 1

Prefer failing startup over accepting a value that quietly breaks the feature — the failure modes here are all silent.

Scope suggestion

Doing all eight at once is more surface than the need justifies. The two worth having first:

1. WarnThreshold — the most volume-sensitive, and the one most likely to be wrong for a given deployment
2. the per-tick caps — the incident-response levers

The retry-envelope-adjacent ones (disabledSnooze, AutoDisableWindow) interact with the GA-frozen 8-attempt/29h21m delivery envelope, so they want more thought before being exposed.

Context

Filed after the webhook health notification work (#852, #860, #862). The values were reviewed and deliberately kept as-is; the objection is that revisiting them later costs a release rather than a config edit.

🤖 Filed with Claude Code